Signal processing method and apparatus and recording medium

ABSTRACT

A signal processor  12  acquires a second signal obtained by detecting a first signal, as a signal of the real world, having a first dimension. The second signal is of a second dimension lower than the first dimension and has distortion relative to the first signal. The signal processor  12  performs signal processing which is based on the second signal to generate a third signal alleviated in distortion as compared to the second signal.

This is a division of application Ser. No. 09/830,858, filed May 1,2001, which is based on International Application PCT/JP00/09421 filedDec. 28, 2000, pursuant to 35 USC 371, and is entitled to the priorityfiling date of Japanese application 11/373782, filed in Japan on Dec.28, 1999, the entirety of which is incorporated herein by reference.

TECHNICAL FIELD

This invention relates to a signal processing method and apparatus andto a recording medium. More particularly, it relates to a signalprocessing method and apparatus and to a recording medium which takesthe difference between the signals as detected by the sensor and thereal world into account.

BACKGROUND ART

Such a technique is widely exploited which detects events in the realworld by sensors and which processes sampling data output by thesensors, such as data associated with pictures, speech, temperature,pressure, acceleration or odor.

For example, a picture obtained on imaging an object moving in front ofa predetermined still background by a video camera is subjected tomotion blurring in case the object is moved at a higher velocity.

For example, a picture obtained on imaging an object moving in front ofa predetermined still background by a video camera employing a CCD issubjected to motion blurring in case the object is moved at a highervelocity. That is, when the real world is detected by a CCD as a sensor,the picture, as sampling data, undergoes distortion.

The conventional practice in suppressing this motion blurring is toincrease the speed of e.g., an electronic shutter to provide for shorterlight exposure time.

However, in raising the shutter speed in this manner, it is necessary toadjust the shutter speed of the video camera before proceeding tophotographing. So, the blurred picture, previously acquired, cannot becorrected to obtain a clear picture.

On the other hand, if an object is moved in front of a stationarybackground, not only motion blurring due to mixing of no other than thepicture of the moving object, but also the mixing of the backgroundpicture and the moving object occurs. In the conventional system, noconsideration is given to detecting the mixing state of the backgroundpicture and the moving object.

Moreover, the information of the real world having the space and thetime axis is acquired by a sensor and made into data. The data acquiredby the sensor is the information obtained on projecting the informationof the real world in the time and space of a lower dimension than thereal world. So, the information obtained on projection is distorted dueto projection. Stated differently, the data output by the sensor isdistorted relative to the information of the real world. Moreover, thedata, distorted by projection, also includes the significant informationfor correcting the distortion.

In the conventional signal processing on the sampling data, acquired bythe sensor, the sampling data obtained by the sensor is deemed to be themost reliable data, such that, in subsequent data processing, such astransmission, recording or reproduction, it has been a sole concern torealize the state of data which is as close to that of the original dataas possible, in consideration of deterioration caused by e.g., datatransmission.

Heretofore, the sampling data output by the sensor is deemed to be themost reliable data, such that no attempt has been made to prepare datahigher in quality than the sampling data or to perform signal processingof extracting the significant information buried by projection.

DISCLOSURE OF THE INVENTION

It is therefore an object of the present invention to provide foradjustment of the amount of motion blurring contained in detectionsignals of a:blurred picture.

It is another object of the present invention to enable detection of amixing ratio indicating the state of mixing of plural objects such as abackground picture and a picture of a moving object.

It is yet another object of the present invention to provide a signalprocessing apparatus in which sampling data output by a sensor may befreed of distortion or the significant information can be extracted fromthe sampling data, for example, to provide for adjustment of the amountof motion blurring contained in the detection signal if the samplingdata is that of picture.

The present invention provides a picture processing apparatus forprocessing picture data made up of a predetermined number of pixel dataacquired by an imaging device having a predetermined number of pixelseach having an integrating effect, the picture processing apparatusincluding processing unit decision means for deciding, based on areainformation specifying a foreground area made up only of foregroundobject components making up a foreground object in the picture data, abackground area made up only of background object components making up abackground object in the picture-data, and a mixed area which is amixture of the foreground object components and the background objectcomponents in the picture data, the mixed area including a coveredbackground area formed at a leading end in a movement direction of theforeground object, and an uncovered background area formed at a trailingend of the foreground object, a processing unit made up of pixel datalying on at least a straight line extending in a direction coincidentwith the direction of movement of the foreground object from an outerend of the covered background area to an outer end of the uncoveredbackground area, centered about the foreground area, normal equationgenerating means for generating a normal equation by setting pixelvalues of pixels in the processing unit decided based on the processingunit and a dividing value which is an unknown dividing value obtained ondividing the foreground object components in the mixed area with apredetermined dividing number, and calculating means for solving thenormal equation by the least square method to generate foreground objectcomponents adjusted for the quantity of movement blurring.

The present invention also provides a picture processing method forprocessing picture data made up of a predetermined number of pixel dataacquired by an imaging device having a predetermined number of pixelseach having an integrating effect, the picture processing methodincluding a processing unit decision step of deciding, based on areainformation specifying a foreground area made up only of foregroundobject components making up a foreground object in the picture data, abackground area made up only of background object components making up abackground object in the picture data, and a mixed area which is amixture of the foreground object components and the background objectcomponents in the picture data, the mixed area including a coveredbackground area formed at a leading end in a movement direction of theforeground object, and an uncovered background area formed at a trailingend of the foreground object, a processing unit made up of pixel datalying on at least a straight line extending in a direction coincidentwith the: direction of movement of the foreground object from an outerend of the covered background area to an outer end of the uncoveredbackground area, centered about the foreground area, a normal equationgenerating step of generating a normal equation by setting pixel valuesof pixels in the processing unit decided based on the processing unitand a dividing value which is an unknown dividing value obtained ondividing the foreground object components in the mixed area with apredetermined dividing number, and a calculating step of solving thenormal equation by the least square method to generate foreground objectcomponents adjusted for the quantity of movement blurring.

The present invention also provides a picture processing program forprocessing picture data made up of a predetermined number of pixel dataacquired by an imaging device having a predetermined number of pixelseach having an integrating effect, the picture processing programincluding a processing unit decision step of deciding, based on areainformation specifying a foreground area made up only of foregroundobject components making up a foreground object in the picture data, abackground area made up only of background object components making up abackground object in the picture data, and a mixed area which is amixture of the foreground object components and the background objectcomponents in the picture data, the mixed area including a coveredbackground area formed at a leading end in a movement direction of theforeground object, and an uncovered background area formed at a trailingend of the foreground object, a processing unit made up of pixel datalying on at least a straight line extending in a direction coincidentwith the direction of movement of the foreground object from an outerend of the covered background area to an outer end of the uncoveredbackground area, centered about the foreground area, a normal equationgenerating step of generating a normal equation by setting pixel valuesof pixels in the processing unit decided based on the processing unitand a dividing value which is an unknown dividing value obtained ondividing the foreground object components in the mixed area with apredetermined dividing number and a calculating step of solving thenormal equation by the least square method to generate foreground objectcomponents adjusted for the quantity of movement blurring.

The present invention also provides a signal processing apparatus forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing apparatus including foreground sample data extracting meansfor extracting the sample data present in detection data before or afterconsidered detection data where there exists considered sample datawhich is the sample data under consideration, as foreground sample datacorresponding to an object proving the foreground in the real world,background sample data extracting means for extracting the sample datapresent in detection data lying after or before the considered detectiondata where there exists considered sample data which is the sample dataunder consideration, as background sample data corresponding to anobject proving the background in the real world and detection means fordetecting a mixing ratio of the considered sample data based on theconsidered sample data, the foreground sample data and the backgroundsample data.

The present invention also provides a signal processing method forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing method including a foreground sample data extracting step ofextracting the sample data present in detection data before or afterconsidered detection data where there exists considered sample datawhich is the sample data under consideration, as foreground sample datacorresponding to an object proving the foreground in the real world, abackground sample data extracting step of extracting the sample datapresent in detection data lying after or before the considered detectiondata where there exists considered sample data which is the sample dataunder consideration, as background sample data corresponding to anobject proving the background in the real world and a detection step ofdetecting a mixing ratio of the considered sample data based on theconsidered sample data, the foreground sample data and the backgroundsample data.

The present invention also provides a signal processing program forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing program including a foreground sample data extracting step ofextracting the sample data present in detection data before or afterconsidered detection data where there exists considered sample datawhich is the sample data under consideration, as foreground sample datacorresponding to an object proving the foreground in the real world, abackground sample data extracting step of extracting the sample datapresent in detection data lying after or before the considered detectiondata where there exists considered sample data which is the sample dataunder consideration, as background sample data corresponding to anobject proving the background in the real world and a detection step ofdetecting a mixing ratio of the considered sample data based on theconsidered sample data, the foreground sample data and the backgroundsample data.

The present invention also provides a signal processing apparatus forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing apparatus including still/movement decision means fordeciding still/movement based on the detection data, and detection meansfor detecting a mixed area containing sample data having plural realworld objects mixed together based on the results of discrimination.

The present invention also provides a signal processing method forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing method including a still/movement decision step of decidingstill/movement based on the detection data, and a detection step ofdetecting a mixed area containing sample data having plural real worldobjects mixed together based on the results of discrimination.

The present invention also provides a signal processing program forprocessing detection data, acquired every predetermined time period by asensor made up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, the signalprocessing program including a still/movement decision step of decidingstill/movement based on the detection data, and a detection step ofdetecting a mixed area containing sample data having plural real worldobjects mixed together based on the results of discrimination.

The present invention also provides a signal processing apparatusincluding means for acquiring second signals of a second dimension byprojecting first signals as real-world signals of a first dimension on asensor and by detecting the mapped signals by the sensor, the firstdimension being lower than the first dimension, and signal processingmeans for extracting the significant information, buried by theprojection from the second signals, by performing signal processingwhich is based on the second signals.

The present invention also provides a recording medium having recordedthereon a computer-readable program, the program including a signalacquisition step of acquiring a second signal by projecting a firstsignal as a real world signal of a first dimension on a sensor anddetecting the so-mapped first signal by the sensor, the signal being ofa second dimension lower than the first dimension, and a signalprocessing step of performing signal processing based on the secondsignal to extract the significant information buried by projection fromthe second signal.

The present invention provides a signal processing apparatus includingsignal acquisition means for acquiring a second signal by detecting afirst signal as a real world signal of a first dimension by a sensor,the signal being of a second dimension lower than the first dimensionand containing distortion with respect to the first signal, and signalprocessing means for performing signal processing on the second signalfor generating a third signal alleviated in distortion as compared tothe second signal.

The present invention also provides a signal processing apparatus forprocessing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the signal processing apparatus including areaspecifying means for specifying a foreground area made up only offoreground object components constituting an foreground object, abackground area made up only of background object componentsconstituting a background object, and a mixed area mixed from theforeground object components and the background object components, mixedarea detection means for detecting a mixing ratio of the foregroundobject components and the background object components at least in themixed area, and separating means for separating the foreground objectand the background object from each other based on the specified resultsby the area specifying means and the mixing ratio.

The present invention also provides a signal processing method forprocessing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the signal processing method including an areaspecifying step of specifying a foreground area, made up only offoreground object components constituting an foreground object, abackground area made up only of background object componentsconstituting a background object, and a mixed area mixed from theforeground object components and the background object components, amixed area detection step of detecting a mixing ratio of the foregroundobject components and the background object components at least in themixed area, and a separating step of separating the foreground objectand the background object from each other based on the specified resultsby the area specifying means and the mixing ratio.

The present invention also provides a recording medium having acomputer-readable program, recorded thereon, the computer-readableprogram including an area specifying step of specifying a foregroundarea, made up only of foreground object components constituting anforeground object, a background area made up only of background objectcomponents constituting a background object, and a mixed area mixed fromthe foreground object components and the background object components, amixed area detection step of detecting a mixing ratio of the foregroundobject components and the background object components at least in themixed area and a separating step of separating the foreground object andthe background object from each other based on the specified results bythe area specifying means and the mixing ratio.

The present invention also provides a signal processing apparatus forprocessing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the signal processing apparatus including areaspecifying means for specifying a foreground area, made up only offoreground object components constituting an foreground object, abackground area made up only of background object componentsconstituting a background object, and a mixed area mixed from theforeground object components and the background object components, andmixing ratio detecting means for detecting a mixing ratio between theforeground object components and the background object components atleast in the mixed area based on the results specified by the areaspecifying means.

The present invention also provides a signal processing method forprocessing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the signal processing method including an areaspecifying step of specifying a foreground area, made up only offoreground object components constituting an foreground object, abackground area made up only of background object componentsconstituting a background object, and a mixed area mixed from theforeground object components and the background object components, and amixing ratio detecting step of detecting a mixing ratio between theforeground object components and the background object components atleast in the mixed area based on the results specified by the areaspecifying means.

The present invention also provides a recording medium having acomputer-readable program recorded thereon, the signal processing methodfor processing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the computer-readable program including anarea specifying step of specifying a foreground area, made up only offoreground object components constituting an foreground object, abackground area made up only of background object componentsconstituting a background object, and a mixed area mixed from theforeground object components and the background object component, and amixing ratio detecting step of detecting a mixing ratio between theforeground object components and the background object components atleast n the mixed area based on the results specified by the areaspecifying means.

The present invention also provides a signal processing apparatus forprocessing a predetermined number of detection signals acquired by asensor made up of a predetermined number of detection elements havingtime integrating effects, the signal processing apparatus includingmixing ratio detecting means for detecting a mixing ratio of foregroundobject components and background object components in a mixed area inwhich said foreground object components constituting a foreground objectand said background object components constituting a background objectare mixed, and separating means for separating the foreground object andthe background object from each other based on the mixing ratio.

The present invention provides a signal processing method for processinga predetermined number of detection signals acquired by a sensor made upof a predetermined number of detection elements having time integratingeffects, the signal processing method including a mixing ratio detectingstep of detecting a mixing ratio of foreground object components andbackground object components in a mixed area in which said foregroundobject components constituting a foreground object and said backgroundobject components constituting a background object are mixed, and aseparating step of separating the foreground object and the backgroundobject from each other based on the mixing ratio.

The present invention also provides a recording medium having recordedthereon a computer-readable program for processing a predeterminednumber of detection signals acquired by a sensor made up of apredetermined number of detection elements having time integratingeffects, the computer-readable program including a mixing ratiodetecting step of detecting a mixing ratio of foreground objectcomponents and background object components in a mixed area in whichsaid foreground object components constituting a foreground object andsaid background object components constituting a background object, anda separating step of separating the foreground object and the backgroundobject from each other based on the mixing ratio.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the principle of the present invention.

FIG. 2 is a block diagram showing an illustrative structure of a systemembodying the present invention.

FIG. 3 is a block diagram showing an illustrative structure of a signalprocessor of FIG. 2.

FIG. 4 is a flowchart for illustrating the operation of the system ofFIG. 2.

FIG. 5 illustrates a typical picture acquired at step S1 of FIG. 4.

FIG. 6 illustrates pixel values of a mixed area.

FIG. 7 illustrates the result of subtracting picture components of thebackground in domains D1 to D3 of FIG. 6.

FIG. 8 illustrates the structure of motion blurring.

FIG. 9 is a flowchart for illustrating another typical processing of thesystem of FIG. 2.

FIG. 10 is a block diagram showing a signal processor 12.

FIG. 11 illustrates the photographing by a sensor.

FIG. 12 illustrates pixel arrangement.

FIG. 13 illustrates the operation of a detection device.

FIG. 14 illustrates a picture obtained on imaging an objectcorresponding to the moving foreground and an object corresponding to astationary background.

FIG. 15 illustrates a background area, a foreground area, a mixed area,a covered background area and an uncovered background area.

FIG. 16 is a diagrammatic view showing pixel values of pixels arrangedin a neighboring relation to another in a row in a picture obtained onimaging an object corresponding to the stationary foreground and apicture obtained on imaging an object corresponding to the stationarybackground, with the pixel values extended in the time axis direction.

FIG. 17 is a diagrammatic view showing pixel values extended in the timeaxis direction, with the time period corresponding to the shutter timeshown split.

FIG. 18 is a diagrammatic view showing pixel values extended in the timeaxis direction, with the time period corresponding to the shutter timeshown split.

FIG. 19 is a diagrammatic view showing pixel values extended in the timeaxis direction, with the time period corresponding to the shutter timeshown split.

FIG. 20 shows extracted pixels of a foreground area, a background areaand a mixed area.

FIG. 21 shows the relation of correspondence between pixels and a modelobtained on expanding the pixel values in the time axis direction.

FIG. 22 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 23 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 24 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 25 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 26 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 27 is a flowchart for illustrating the processing for adjusting theamount of the motion blurring.

FIG. 28 is a block diagram showing an illustrative structure of an areaspecifying unit 103.

FIG. 29 illustrates a picture as an object corresponding to theforeground is being moved.

FIG. 30 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 31 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 32 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 33 illustrates a condition for areal decision.

FIGS. 34A, 34B, 34C and 34D illustrate the results of identification ofareas of the area specifying unit 103.

FIG. 35 illustrates the results of identification of areas of the areaspecifying unit 103.

FIG. 36 is a flowchart for illustrating the processing for arealidentification.

FIG. 37 is a block diagram showing an illustrative structure of a mixingratio calculating unit 104.

FIG. 38 shows a typical ideal mixing ratio α.

FIG. 39 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 40 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 41 illustrates the approximation exploiting the correlation of theforeground components.

FIG. 42 illustrates the relation between C, N and P.

FIG. 43 is a block diagram showing the structure of an estimated mixingratio processor 201.

FIG. 44 shows a typical estimated mixing ratio.

FIG. 45 is a block diagram showing a modified structure of the mixingratio calculating unit 104.

FIG. 46 is a flowchart for illustrating the processing for calculatingthe estimated mixing ratio.

FIG. 47 is a flowchart for illustrating the processing for the operationof the estimated mixing ratio.

FIG. 48 is a block diagram showing an illustrative structure of aforeground/background separating unit 105.

FIGS. 49A and 49B show an input picture, a foreground component pictureand a background component picture.

FIG. 50 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 51 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 52 is a diagrammatic view showing pixel values developed in thetime axis direction and showing the time period corresponding to theshutter period shown split.

FIG. 53 is a block diagram showing an illustrative structure of aseparator 251.

FIGS. 54A and 54B illustrate typical examples of a foreground componentpicture and a background component picture as separated from each other.

FIG. 55 is a flowchart for illustrating the processing for separatingthe foreground and the background from each other.

FIG. 56 is a block diagram showing an illustrative structure of a motionblurring adjustment unit 106.

FIG. 57 illustrating a processing unit.

FIG. 58 is a diagrammatic view showing pixel values of a foregroundcomponent picture developed in the time axis direction and showing thetime period corresponding to the shutter period shown split.

FIG. 59 is a diagrammatic view showing pixel values of a foregroundcomponent picture developed in the time axis direction and showing thetime period corresponding to the shutter period shown split.

FIG. 60 is a diagrammatic view showing pixel values of a foregroundcomponent picture developed in the time axis direction and showing thetime period corresponding to the shutter period shown split.

FIG. 61 is a diagrammatic view showing pixel values of a foregroundcomponent picture developed in the time axis direction and showing thetime period corresponding to the shutter period shown split.

FIG. 62 shows a modified structure of the motion blurring adjustmentunit 106.

FIG. 63 is a flowchart for illustrating the processing for adjusting theamount of motion blurring contained in the foreground component picture

FIG. 64 is a block diagram showing a modified structure of the functionof a signal processor 12.

FIG. 65 shows the structure of a synthesis unit 371.

FIG. 66 is a block diagram showing another modified structure of thefunction of the signal processor 12.

FIG. 67 is a block diagram showing the structure of a mixing ratiocalculating unit 401.

FIG. 68 is a block diagram showing the structure of aforeground/background separating unit 402.

FIG. 69 is a block diagram showing a further modified structure of thefunction of the signal processor 12.

FIG. 70 illustrates the structure of a synthesis unit 431.

FIG. 71 shows another illustrative structure of a signal processingapparatus according to the present invention.

FIG. 72 is a flowchart for illustrating the processing for adjusting theamount of motion blurring by a signal processor 452.

FIG. 73 shows an illustrative structure of a signal processing apparatusaccording to the present invention.

FIG. 74 shows a structure of a pressure area sensor 501.

FIG. 75 illustrates the load applied to the pressure area sensor 501.

FIG. 76 illustrates typical weight data output by the pressure areasensor 501.

FIG. 77 is a flowchart for illustrating the load calculating processingexecuted by a signal processor 502.

FIG. 78 is a block diagram showing the structure of generating a picturehaving an increased number of pixels per frame, as another function ofthe signal processor 12.

FIG. 79 illustrates pixel arrangement and an area corresponding to apixel doubled in horizontal density.

FIG. 80 illustrates a picture component of a picture corresponding tolight input to areas A to r.

FIGS. 81A, 81B, 81C and 81D illustrates calculation of picturecomponents corresponding to two areas of a pixel.

FIG. 82 shows a typical input picture.

FIG. 83 shows a typical double horizontal density picture.

FIG. 84 shows a typical double vertical density picture.

FIG. 85 shows a double density picture.

FIG. 86 is a flowchart for illustrating the processing for generating adouble density picture by a signal processor 12 shown in FIG. 78.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 shows the principle of the present invention. As may be seen inFIG. 1, a first signal, as the information of a real world 1 having thespatial axis and the temporal axis, is acquired by a sensor 2, and ismade into data. A detection signal, as data 3 acquired by the sensor 2,is the information obtained on projecting the information of the realworld 1 on a time space of a lower dimension than in the real world 1.Therefore, the information, resulting from the projection, containsdistortion ascribable to projection. Stated differently, the data 3,output by the sensor 2, is distorted relative to the information of thereal world 1. Moreover, the data 3, thus distorted as a result ofprojection, also includes the significant information usable orcorrecting the distortion.

Thus, according to the present invention, the data output by the sensor2 is processed by a signal processor 4, whereby the distortion isremoved, reduced or adjusted. Alternatively, the data output by thesensor 2 is processed by the signal processor 4 to extract thesignificant information.

FIG. 2 shows an illustrative structure of a signal processing apparatusaccording to the present invention The sensor 1 is comprised e.g., of avideo camera, which photographs a picture of the real world to outputthe resulting picture data to the signal processor 12. The signalprocessor is comprised e.g., of a personal computer for processing thedata input from the sensor 11, adjusting the amount of distortionproduced by projection, specifying an area containing the significantinformation buried by the projection, extracting the significantinformation from a specified area and for processing the input databased on the extracted significant information.

The significant information may, for example, be a mixing ratio, aslater explained.

Meanwhile, the information indicating an area containing the significantinformation buried by the projection may also be deemed to be thesignificant information. Here, the areal information, as laterexplained, corresponds to the significant information.

The signal processor 12 is configured as shown for example in FIG. 3. ACPU (central processing unit) 21 executes various processing operationsin accordance with a program stored in a ROM (read-only memory) 22 or ina storage unit 28. In a RAM (random access memory) 23, the programexecuted by the CPU 21 or data are stored as necessary. The CPU 21, ROM22 and the RAM 23 are interconnected over a bus 24.

To the CPU 21 is connected an input/output interface 25 over a bus 24.To the input/output interface 25 are connected an input unit 26,comprised of a keyboard, a mouse and a microphone, and an output unit27, comprised of a display and a speaker. The CPU 21 executes variousprocessing operations responsive to commands input from the input unit26. The CPU 21 outputs a picture, speech and so forth, obtained onprocessing, to the output unit 27.

The storage unit 28, connected to the input/output interface 25, isconstituted e.g., by a hard disc, for storing the program executed bythe CPU 21 and a variety of data. A communication unit 29 communicatewith external equipment over a network, such as Internet. In the presentembodiment, the communication unit 29 operates for acquiring an outputof the sensor 11.

The program may also be acquired over the communication unit 29 forstorage in the storage unit 28.

A driver 30 connected to the input/output interface 25 drives a magneticdisc 51, an optical disc 52, a magneto-optical disc 53 or asemiconductor memory 54, to acquire the program and data recordedtherein, when these devices are connected thereto. The program and thedata, thus acquired, are transferred to the storage unit 28, asnecessary, for storage therein.

Referring to the flowchart of FIG. 4, the operation performed by thesignal processing apparatus based on the program stored in the storageunit 28, is explained. First, at step S1, a picture of an object,acquired by the sensor 11, is acquired through e.g., the communicationunit 29. The CPU 21 of the signal processor 12 sends the acquiredpicture data to the storage unit 28 for storage therein.

FIG. 5 shows a picture associated with the so-acquired picture data. Thepicture, shown in this embodiment, is comprised of a foreground 62arranged ahead of a background 61. The foreground here is a toy planemoving at a predetermined speed ahead of the still background 61 towardsright in the drawing. The result is that the picture of the foreground62 is a picture subjected to so-called motion blurring. Conversely, thepicture of the background 61 is stationary and hence is a clear picturefree of motion blurring. A mixed area 63 is a picture comprised of amixture of an object which is the background 61 and an object which isthe foreground 62.

Then, at step S2, the CPU 21 detects the mixed area of the objects. Inthe embodiment of FIG. 5, the mixed area 63 is detected as an area ofthe mixture of the two objects.

The CPU 21 at step S3 decides whether or not the objects are mixed. Ifthe objects are not mixed, that is if there is no mixed area 63, thepicture is not what is to be processed by the present informationprocessing apparatus and hence the processing is finished.

If conversely a decision is made at step S3 that the objects are mixed,the CPU 21 proceeds to step S4 to find the object mixing ratio in thedetected mixed area. The mixing ratio may be found by finding the motionvector of the foreground 21 relative to the background 61 and byfitting, from the motion vector, so that the mixing ratio in the mixedarea 63 will be changed in a range from 0 to 1. At step S5, the CPU 21performs the processing of separating the objects in the mixed area 63where plural objects are mixed together, based on the so-found mixingratio.

The above-described processing is explained in further detail, taking apicture of FIG. 5 as an example. If picture data on one line of aportion 63A on the right end of the mixed area 63 of FIG. 5 is plotted,the result is as shown in FIG. 6, in which the abscissa denotesX-coordinates (coordinates in the horizontal direction in FIG. 5) andthe ordinate denotes pixel values on the X-coordinates.

A curve L1 denotes pixel values on a line of a first timing, whilst acurve L2 denotes pixel values on another line of the next timing.Similarly, curves Light reflecting layer 3 and L4 denote pixel values oflines of the sequentially consecutive timings. Stated differently, FIG.6 shows changes in the pixel values on associated lines at the fourconsecutive timings.

The curve Li shows the state in the first timing in which state theforeground 62 has not yet been imaged. So, the curve L1 representspixels of the foreground 61.

On the curve L1, the pixel value is approximately 75 in the vicinity ofthe X-coordinate 140, and is increased to approximately 130 at theX-coordinate 145. The pixel value then is lowered and is approximately120 in the vicinity of the X-coordinate 149. As the X-coordinate isincreased, the pixel value is again increased and reaches approximately160 in the vicinity of the X-coordinate 154. The pixel value then isagain lowered and reaches approximately 130 in the vicinity of theX-coordinate 162. Then, in the vicinity of the X-coordinate of 165, thepixel value is approximately 180 and, in the vicinity of theX-coordinate of 170, the pixel value is again lowered to approximately125. Then, in the vicinity of the X-coordinate of 172, the pixel valueis increased to approximately 175 and, in the vicinity of theX-coordinate of 178, the pixel value is lowered to approximately 60.Subsequently, the pixel value is slightly fluctuated between 60 and 80in a domain of the X-coordinates of from 178 to 195. In theX-coordinates on the further right side of approximately. 195, the pixelvalue is again increased to approximately 160.

As for the curve L2 of the next frame, the pixel value is constant atapproximately 200 up to the pixel value of 145. The pixel value then isgradually lowered in a range from the X-coordinate of 145 to theY-coordinate of 160, at which Y-coordinate value the pixel value isapproximately 125. The curve then undergoes changes in a manner similarto those of the curve L1.

The pixel value of the curve Light reflecting layer 3 is substantiallyconstant at 200 up to the vicinity of the X-coordinate 158 and is thenlowered to approximately 164 at the X-coordinate 164, after which it isincreased to approximately 190. The curve then undergoes changes in amanner similar to those of the curve L1.

The pixel value of the curve L4 is constant at approximately 200 fromthe vicinity of the X-coordinate of 140 up to the vicinity of theX-coordinate 170, and is abruptly lowered from the vicinity of theX-coordinate of 170 up to the vicinity of the X-coordinate 180, with thepixel value in the vicinity of the X-coordinate of 170 beingapproximately 70. The curve then undergoes changes in a manner similarto those of the curve L1.

These changes in the pixel values of the curves L2 to L4 are ascribableto the fact that, while the picture of only the background 61 exists inthe state of the curve L1, the picture of the foreground 62 is graduallyincreased with the movement of the picture of the foreground 62, that iswith lapse of time.

Specifically, as may be seen from comparison of the curve L1 and thecurve L2 of the next following timing, the values of the curves L2 to L4are substantially equal in values up to the vicinity of the X-coordinateof 147. Beginning from the vicinity of the X-coordinate 147, the valuesof the curve L2 differ from those of the curves Light reflecting layer3, L4, becoming equal to the values of the curve L1 in vicinity of theX-coordinate 159. Subsequently, the pixel values of the curve L2 areapproximately equal to those in the curve L1. That is, the values of thecurve L2 in an area R1 corresponding to a domain D1 from an X-coordinate146 to an X-coordinate 159 indicate that the foremost part of theforeground 62 has been moved from the left end to the right end of thedomain D1 during one unit period.

Similarly, the pixel values of the curve Light reflecting layer 3 of thenext timing in an area Rigid substrate 2 corresponding to a domain D2from an X-coordinate 159 to an X-coordinate 172 indicate that theforemost part of the foreground 62 has been moved in the interim. Thepixel values of the curve L4 of the next timing in an area R3corresponding to a domain D3 from the X-coordinate 172 to anX-coordinate 184 indicate that the foremost part of the foreground 62has been moved in the interim.

So, if the pixel values of the curve L1, weighted on the basis of amixing ratio of the foreground 62 to the background 61, are subtractedfrom the pixel values of the curve L2, a curve L11 shown in FIG. 7 isobtained. This curve L11, tantamount to subtraction of the valuescorresponding to background 61 from the pixels of the foreground 62 inthe mixed area 63, represents a picture of the foreground on thebackground having the pixel value of 0. Meanwhile, in FIG. 7, theabscissa and the ordinate denote the position and the pixel values ofthe extracted foreground, respectively. As for the position, the leftand the right end correspond to the left and right ends in the domain D1in FIG. 6, respectively.

Similarly, if, in the domain D2 of FIG. 6, the pixel values of the curveL1, weighted by the mixing ratio, are subtracted from the pixel valuesof the curve Light reflecting layer 3, a curve L12 in FIG. 7 isobtained, whereas, if, in the domain D3 of FIG. 6, the pixel values ofthe curve L1, weighted by the mixing ratio, are subtracted from thecurve L4, a curve L13 in FIG. 7 is obtained. The curves L12, L13 aresubstantially coincident with the curve L11, as shown in FIG. 7. Thisindicates that the foreground 62 is moving at an approximately equalspeed during the three timing unit periods, and that the blackbackground, that is the foreground pixel values on the background havingthe pixel value of 0, has been obtained correctly by weightedsubtraction.

If the above-described operation is explained in connection with pixelsby referring to FIG. 8, in which the abscissa denotes the X-coordinateof a portion 63A, with the ordinate denoting the time axis directingfrom above towards below. Since the amount of movement is 5 in thepresent embodiment, light exposure is made within the time interval oft1 to t5 (within the shutter time). In FIG. 8, b1 to bf denote pixelvalues of the respective pixels of the background 61 and A1 to A6 denotepixel values of the foreground 62.

That is, the pixels A1 to A6 of the foreground 62 appear at thepositions of the pixels b3 to b8 of the background 61, with the pixelsA1 to A6 of the foreground 62 moving rightwards at the timing t2 by onepixel, that is to the position of the pixels b4 to b9 of the background61.

In similar manner, the pixels A1 to A6 of the foreground 62 aresequentially moved rightwards at a pitch of one pixel as time elapsesfrom timing t3 to timing t5. In this case, the pixel values y1 to yf,obtained on averaging the pixels of the respective lines of the timingst1 to t5, constitute pixels obtained on imaging, that is pixelsexhibiting motion blurring, with the values being represented by thefollowing equations:

$\begin{matrix}{y_{3} = {{\frac{1}{5} \cdot a_{1}} + {\frac{4}{5} \cdot b_{3}}}} & (1) \\{y_{4} = {{\frac{1}{5} \cdot \left( {a_{1} + a_{2}} \right)} + {\frac{3}{5} \cdot b_{4}}}} & (2) \\{y_{5} = {{\frac{1}{5} \cdot \left( {a_{1} + a_{2} + a_{3}} \right)} + {\frac{2}{5} \cdot b_{5}}}} & (3) \\{y_{6} = {{\frac{1}{5} \cdot \left( {a_{1} + a_{2} + a_{3} + a_{4}} \right)} + {\frac{1}{5} \cdot b_{6}}}} & (4) \\{y_{7} = {\frac{1}{5} \cdot \left( {a_{1} + a_{2} + a_{3} + a_{4} + a_{5}} \right)}} & (5) \\{y_{8} = {\frac{1}{5} \cdot \left( {a_{1} + a_{2} + a_{3} + a_{4} + a_{5} + a_{6}} \right)}} & (6) \\{y_{9} = {{\frac{1}{5} \cdot \left( {a_{3} + a_{4} + a_{5} + a_{6}} \right)} + {\frac{1}{5} \cdot b_{9}}}} & (7) \\{y_{a} = {{\frac{1}{5} \cdot \left( {a_{4} + a_{5} + a_{6}} \right)} + {\frac{2}{5} \cdot b_{a}}}} & (8) \\{y_{b} = {{\frac{1}{5} \cdot \left( {a_{5} + a_{6}} \right)} + {\frac{3}{5} \cdot b_{b}}}} & (9) \\{y_{c} = {{\frac{1}{5} \cdot a_{6}} + {\frac{4}{5} \cdot b_{c}}}} & (10)\end{matrix}$Meanwhile, y1, y2, yd, ye and yf are equal to background pixels b1, b2,bd, be and bf, respectively.

If pixels b1 to bf of the background are removed, the background 61 andthe foreground 62 in the mixed area 63 can be separated from each other.That is, plural objects can be separated from one another. Moreover, thebackground pixels b1 to bf can be found by solving the above equations,using, for example, the least square method, by assuming the backgroundpixels b1 to bf to be known such as by employing the pixel values of thefore and aft shutter time (frame). This gives a foreground picture freedof the motion blurring. In this manner, distortion caused by projectionin the information of the real world can be reduced to create a clearpicture such as by processing for resolution creation.

In FIG. 4, it is the deterministic processing that is executed, that is,the previous processing is used as basis and the next followingprocessing is executed on the assumption that the result of the previousprocessing is just. Alternatively, statistic processing is alsopossible, as now explained with reference to illustrative processingshown in FIG. 9.

Specifically, when carrying out the statistic processing, the CPU 21acquires picture data at step S21. This processing is similar to thatperformed at step S1 in FIG. 4.

Next, st step S22, the CPU 21 performs the processing of finding themixing ratio of the foreground an the background from the picture dataobtained at step S21. At step S23, the CPU 21 executes the processing ofseparating the foreground and the background based on the mixing ratiofound at step S22.

If the statistic processing is used, the processing of deciding whetheror not the boundary of an object exists, such as that at step S23 ofFIG. 4, is unnecessary, thus enabling the foreground and the backgroundto be separated from each other more expeditiously.

The foregoing shows the manner as to how a clear picture of theforeground 62 can be separated and extracted from the motion-blurredpicture obtained on photographing a picture of the foreground 62 movingahead of the background 61.

A more specified embodiment of a signal processing apparatus foridentifying an area having the significant information buried therein orextracting the so-buried significant information from data acquired fromthe sensor by the deterministic processing is now explained. In thefollowing embodiment, a CCD line sensor or a CCR area sensor correspondsto the sensor, while the areal information or the mixing ratiocorresponds to the significant information and the mixing of theforeground and the background or the motion blurring corresponds todistortion.

FIG. 10 is a block diagram showing the signal processor 12.

Meanwhile, it does not matter whether the respective functions of thesignal processor 12 are to be implemented by hardware or by software.That is, the block diagrams of the present specification may be deemedto be a hardware block diagram or a functional software block diagram.

It is noted that the motion blurring means distortion contained in amoving object, which distortion is produced by the movement of an objectin the real world being imaged and by imaging characteristics proper tothe sensor 11.

In the present specification, the picture corresponding to an object inthe real world is called a picture object.

An input picture, supplied to the signal processor 12, is furnished toan object extraction unit 101, an area specifying unit 103, a mixingratio calculating unit 104 and a foreground background separating unit105.

The object extraction unit 101 roughly extracts a picture objectcorresponding to a foreground object contained in the input picture tosend the extracted picture object to a motion detection unit 102. Theobject extraction unit 101 detects the contour of the picture objectcorresponding to the foreground object contained in the input picture toroughly extract the picture object corresponding to the foregroundobject.

The object extraction unit 101 roughly extracts the picture objectcorresponding to the foreground object contained in the input picture toroute the extracted picture object to the motion detection unit 102. Theobject extraction unit 101 roughly extracts the picture objectcorresponding to the background object, based on the difference betweenthe input picture and the picture object corresponding to the extractedforeground object.

It is also possible for the object extraction unit 101 to roughlyextract the picture object corresponding to the foreground object andthe picture object corresponding to the background object based on thedifference between the background picture stored in an internalbackground memory and the input picture.

The motion detection unit 102 computes the motion vector of the pictureobject corresponding to the roughly extracted foreground, by techniquessuch as block matching method, gradient method, phase correlation methodor the Pel-Recursive method, to route the motion vector so calculatedand the position information of the motion vector (the informationspecifying the position of the pixel corresponding to the motion vector)to the motion blurring adjustment unit 106.

In the motion vector, output by the motion detection unit 102, there iscontained the information corresponding to a movement quantity v.

It is also possible for the motion detection unit 102 to output thepicture object based motion vector, along with the pixel positioninformation specifying the pixels for the picture object, to the motionblurring adjustment unit 106.

The movement quantity v is a value for representing position changes ofpicture corresponding to a moving object in terms of a pixel-to-pixelinterval as unit. For example, if a picture of an object correspondingto the foreground is moved so as to be displayed at a position offset byfour pixels in a frame with respect to a directly previous frame, themovement quantity v of the object corresponding to the foreground is 4.

Meanwhile, the object extraction unit 101 and the motion detection unit102 are used when the quantity of the motion blurring associated with amoving object is adjusted in the motion blurring adjustment unit 106.

The area specifying unit 103 sends the information specifying each pixelof an input picture to one of the foreground area, a background area ora mixed area and for indicating to which of the foreground area,background area and the mixed area belong the pixels, from pixel topixel, to the mixing ratio calculating unit 104, foreground/backgroundseparating unit 105 and to the motion blurring adjustment unit 106. Theaforementioned information is referred to below as the area information.

The mixing ratio calculating unit 104 calculates the mixing ratio forpixels contained in the mixed area 63, based on the input picture andthe area information supplied from the area specifying unit 103, toroute the so-calculated mixing ratio to the foreground/backgroundseparating unit 105. This mixing ratio is referred to below as a mixingratio α.

The mixing ratio α indicates the proportion in the pixel value of thecomponents of a picture corresponding to the background object, asindicated by an equation (13) to be described later. These componentsare also referred to below as the background components.

The foreground/background separating unit 105 separates the inputpicture into a foreground component picture, made up only of a picturecomponent associated with the foreground, also referred to below asforeground components, and a background component picture, composed onlyof background components, based on the area information supplied fromthe area specifying unit 103, and on the mixing ratio α supplied fromthe mixing ratio calculating unit 104, to route the foreground componentpicture to the motion blurring adjustment unit 106 and to the selectionunit 107. The separated foreground component picture may also be anultimate output. It is possible to realize the foreground and thebackground more accurate than those obtained in the conventional systemin which only the foreground and the background can be specified withouttaking the conventional mixed area into consideration.

The motion blurring adjustment unit 106 decides a processing unit,indicating one or more pixels contained in the foreground componentpicture, based on the movement quantity v as found from the motionvector and on the area information. The processing unit is data forspecifying a set of pixels to be processed for adjusting the quantity ofthe motion blurring.

The motion blurring adjustment unit 106 adjusts the quantity of themotion blurring contained in the foreground component picture, such asby removing the motion blurring contained in the foreground componentpicture, decreasing the quantity of the motion blurring or increasingthe quantity of the motion blurring, based on the motion blurringadjusting quantity input to the signal processor 12, foregroundcomponent picture supplied from the foreground/background separatingunit 105, the motion vector supplied from the motion detection unit 102,along with the corresponding position information, and on the processingunit, to output the foreground component picture, adjusted for thequantity of the motion blurring, to the selection unit 107. The motionvector with its position information may not be used, if so desired.

The selection unit 107 selects one of the foreground component picturesupplied from the foreground/background separating unit 105 and theforeground component picture from the motion blurring adjustment unit106, adjusted as to the motion blurring quantity, to output the selectedforeground component picture.

Referring to FIGS. 11 to 26, an input picture sent to the signalprocessor 12 is explained.

FIG. 11 illustrates imaging by a sensor 11 constituted by a CCD videocamera provided with a CCD (charge coupled device) which is a solidstate imaging device. An object corresponding to the foreground in thereal world is moved between the object of the background in the realworld an the sensor 11 e.g., horizontally from left to right.

The sensor 11 images an object corresponding to the foreground alongwith the object corresponding to the background. The sensor 11 outputsthe photographed picture on the frame basis. For example, the sensor 11outputs a picture of 30 frames per sec. The exposure time of the sensor11 may be set to 1/30 sec. The exposure time is the time which elapsessince the start of conversion of light input to the sensor 11 intoelectrical charges until the end of the conversion of the input lightinto electrical charges. This exposure time is sometimes referred tobelow as the shutter time.

Referring to FIG. 12, showing pixel arrangement, A to I denoteindividual pixels. The pixels are arranged in a plane corresponding to apicture. A detection element associated with one pixel is arranged onthe sensor 11. When the sensor 11 photographs a picture, one detectionelement outputs a pixel value associated with one pixel belonging to thepicture. For example, the position of the detection device along theX-direction corresponds to the position on the picture in the transversedirection, whilst that along the Y-direction corresponds to the positionon the picture in the longitudinal direction.

Referring to FIG. 13, a detection device, such as the CCD, converts theinput light into electrical charges, during the time corresponding tothe shutter time, to store the as-converted electrical charges. Thequantity of the electrical charges is approximately equal to theintensity of the input light and to the time during which the light isinput. The detection device sums the electrical charges, converted fromthe input light, to the electrical charges, already stored, during thetime corresponding to the shutter time. That is, the detection deviceintegrates the input light during the time corresponding to the shuttertime to accumulate electrical charges in an amount corresponding to theintegrated light. The detection device is said to have an integratingeffect with respect to time.

The charges accumulated in the detection device are converted into anelectrical voltage by a circuit, not shown. The voltage, in turn, isconverted into a pixel value, such as digital data, which is output. So,the individual pixels, output by the sensor 11, are of a value mapped toa one-dimensional space, which is the result of integration with respectto the shutter time of a spatially extended portion of an objectcorresponding to the foreground or the background.

By such accumulating operation of the sensor 11, the signal processor 12extracts the significant information buried in the output signal, suchas the mixing ratio α. The signal processor 12 adjusts the quantity ofdistortion caused by the mixing of no other than the foreground pictureobject, for example, the quantity of the motion blurring. The signalprocessor 12 also adjusts the quantity of the distortion produced by themixing of the foreground picture object with the background pictureobject.

FIG. 14 illustrates a picture obtained on imaging an objectcorresponding to a moving foreground and an object corresponding to astill background. FIG. 14A shows a picture obtained on imaging an objectcorresponding to the moving foreground and an object corresponding tothe still background. In an embodiment shown in FIG. 14A, the objectcorresponding to the foreground is moving horizontally from left towardsright relative to the picture.

FIG. 14B is a diagrammatic view showing pixel values, corresponding to aline of the picture shown in FIG. 14A, as extended along the time axis.The transverse direction of FIG. 14B corresponds to the spatialdirection X of FIG. 14A.

The pixels of the background area are constituted solely by thebackground components, that is components of a picture corresponding toa background object. The pixels of the: foreground area are constitutedsolely by the foreground components, that is components of a picturecorresponding to a foreground.

The pixels of the mixed area are constituted from the background andforeground components. The mixed area, the pixel values of which areconstituted from the background components and the foregroundcomponents, may be said to be a distorted area. The mixed area isfurther classified into a covered background area and an uncoveredbackground area.

The covered background area is a portion of the mixed area in registerwith the foremost part along the proceeding direction of the foregroundand is an area in which the background component is hidden by theforeground with lapse of time.

On the other hand, the uncovered background area is a portion of themixed area in register with the rear part along the proceeding directionof the foreground and is an area in which the background componentpresents itself with lapse of time.

A picture comprised of the foreground area, background area, a coveredbackground area or the uncovered background area is input as an inputpicture to the area specifying unit 103, mixing ratio calculating unit104 and to the foreground/background separating unit 105.

FIG. 15 illustrates the background area, foreground area, mixed area,covered background area and the uncovered background area, as describedabove. In relation to the picture shown in FIG. 14, the background areais a still portion, the foreground area is a moving portion, the coveredbackground area of the mixed area is an area where the picture ischanged from the background to the foreground, and the uncoveredbackground area of the mixed area is an area where the picture ischanged from the foreground to the background.

FIG. 16 diagrammatically shows pixel values of neighboring pixels in arow in a photographed picture of an object corresponding to a stillforeground and an object corresponding to a sill background, with thepixel values shown developed along the temporal axis direction. As theneighboring pixels, arranged in a row, it is possible to select pixelsarranged on a line of a picture.

The pixel values of F01 to F04, shown in FIG. 16, are those of pixels ofthe object of the still foreground. The pixel values of B01 to B04,shown in FIG. 16, are those of pixels of the object of the stillbackground.

In FIG. 16, time elapses from above towards below. The position of anupper side of a rectangle in FIG. 16 corresponds to the time the sensor11 begins converting the input light into electrical charges, while thatof the rectangle in FIG. 16 corresponds to the time the sensor 11finishes the conversion of the input light into electrical charges. Thatis, the distance from the upper to the lower sides of the rectangle ofFIG. 16 corresponds to the shutter time.

In the following description, it is assumed that the shutter time isequal to the frame interval.

The transverse direction in FIG. 16 corresponds to the spatial directionX, explained with reference to FIG. 14. More specifically, the distancefrom the left side of a rectangle “F01” to the right side of a rectangle“B04” in FIG. 16 is eight times the pixel pitch, that is the span of theeight consecutive pixels.

If the foreground and the background object are still, the light inputto the sensor 11 is not changed during the time corresponding to theshutter time.

The time span corresponding to the shutter time is split into two ormore equal time periods. For example, if the number of times of thevirtual splitting is four, the diagram of FIG. 16 may be represented asthe diagram of FIG. 17. The number of times of the virtual splitting isset in association with e.g., the movement quantity v in the shuttertime of the object corresponding to the foreground. For example, if themovement quantity v is four, the number of times of the virtualsplitting is 4, with the time span corresponding to the shutter timebeing then split into four.

The uppermost row in the drawing corresponds to the first split timeperiod since the time of shutter opening. The second row corresponds tothe second split time period since the time of shutter opening. Thethird row corresponds to the third split time period since the time ofshutter opening, whilst the fourth row corresponds to the fourth splittime period since the time of shutter opening.

The shutter rime split in association with the movement quantity v isalso called the shutter time/v hereinbelow.

When the object corresponding to the foreground is at a standstill, thelight input to the sensor 11 is not changed. So, the foregroundcomponent F01/v is equal to the pixel value F01 divided by the number oftimes of the virtual splitting. Similarly, when the object correspondingto the foreground is at a standstill, the foreground component F02/v isequal to the pixel value F02 divided by the number of times of thevirtual splitting, whilst the foreground component F03/v is equal to thepixel value F03 divided by the number of times of the virtual splittingand the foreground component F04/v is equal to the pixel value. F04divided by the number of times of the virtual splitting.

When the object corresponding to the background is at a standstill, thelight incident on the sensor 11 is not changed. So, the backgroundcomponent B01/v is equal to the pixel value B01 divided by the number oftimes of the virtual splitting. Similarly, when the object correspondingto the background is at a standstill, the background component B02/v isequal to the pixel value B02 divided by the number of times of thevirtual splitting, whilst the background component B03/v is equal to thepixel value. B03 divided by the number of times of the virtual splittingand the background component B04/v is equal to the pixel value B04divided by the number of times of the virtual splitting.

That is, when the object corresponding to the foreground is at astandstill, the light corresponding to the foreground input to thesensor 11 during the time corresponding to the shutter time remainsunchanged. So, the first foreground component F01v, corresponding to theshutter time/v, as from the shutter opening, the second foregroundcomponent F01v, corresponding to the shutter time/v, as from the shutteropening, the third foreground component F01v, corresponding to theshutter time/v, as from the shutter opening and the fourth foregroundcomponent F01v, corresponding to the shutter time/v, as from the shutteropening, are of equal values. The above for F01/v holds for F02/v toF04/v as well.

When the object corresponding to the background is at a standstill, thelight corresponding to the background object input to the sensor 11during the time corresponding to the shutter time remains unchanged. So,the first background component B01v, corresponding to the shuttertime/v, as from the shutter opening, the second background componentB01v, corresponding to the shutter time/v, as from the shutter opening,the third background component B01v, corresponding to the shuttertime/v, as from the shutter opening and the fourth background componentB01v, corresponding to the shutter time/v, as from the shutter opening,are of equal values. The above for B01/v holds for B02/v to B04/v aswell.

In the following description, it is assumed that the objectcorresponding to the foreground is moving, with the object correspondingto the background being at a standstill.

FIG. 18 diagrammatically shows pixel values of pixels arranged on a lineincluding the covered background area when the object corresponding tothe foreground is moving towards right in the drawing, with the pixelvalues being shown developed in the time axis direction. In FIG. 18, themovement quantity v of the foreground is 4. Since one frame is of shortduration, it may be assumed that the object corresponding to theforeground is a rigid body and moving at an equal speed. In FIG. 18, thepicture of the object corresponding to the foreground is moved so as tobe displayed four pixels rightwards in a frame next to a directlyprevious reference frame.

In FIG. 18, the leftmost to fourth left pixels belong to the foregroundarea. In FIG. 18, fifth left to seventh left pixels in FIG. 18 belong tothe mixed area which is the covered background area. In FIG. 18, therightmost pixel belongs to the background area.

Since the object corresponding to the foreground is moved to hide theobject corresponding to the background, as time elapses, the componentscontained in the pixel values of the pixels belonging to the coveredbackground area are switched from the background component picture tothe foreground component picture at a certain time point of the timeperiod corresponding to the shutter time.

For example, the pixel value M, shown with a bold line frame in FIG. 18,is represented by the equation (11):M=B02/v+B02/v+F07/v+F06/v  (11).

For example, the pixel value M, shown with a bold line frame in FIG. 18,contains the background component corresponding to one shutter time/vand the foreground component corresponding to three shutter time/v, themixing ratio α of the fifth left pixel is ¼. The sixth left pixelcontains the background component corresponding to two shutter time/vand the foreground component corresponding to two shutter time/v, so themixing ratio α is ½. The seventh left pixel contains the backgroundcomponent corresponding to three shutter time/v and the foregroundcomponent corresponding to one shutter time/v, so the mixing ratio α is¾.

Since the object corresponding to the foreground is a rigid body, suchthat the foreground picture is moved at an equal speed so as to bedisplayed four pixels towards right in the next frame, the firstforeground component F07/v of the fourth left pixel in FIG. 18, with thefirst shutter time/v since the time of shutter opening, is equal to thesecond foreground component of the fifth left pixel in FIG. 18corresponding to the second shutter time/v since the time of shutteropening. Similarly, the foreground component F07/v is equal to theforeground component of the sixth left pixel in FIG. 18 corresponding tothe third shutter time/v since the time of shutter opening and to theforeground component of the seventh left pixel in FIG. 18 correspondingto the fourth shutter time/v since the time of shutter opening.

Since the object corresponding to the foreground is a rigid body, suchthat the foreground picture is moved at an equal speed so as to bedisplayed four pixels towards right in the next frame, the firstforeground component F06/v of the third left pixel in FIG. 18, with thefirst shutter time/v since the time of shutter opening, is equal to thesecond foreground component of the fourth left pixel in FIG. 18corresponding to the second shutter time/v since the time of shutteropening. Similarly, the foreground component F06/v is equal to theforeground component of the fifth left pixel in FIG. 18 corresponding tothe third shutter time/v since the time of shutter opening and to theforeground component of the sixth left pixel in FIG. 18 corresponding tothe fourth shutter time/v since the time of shutter opening.

Since the object corresponding to the foreground is a rigid body, suchthat the foreground picture is moved at an equal speed so as to bedisplayed four pixels towards right in the next frame, the firstforeground component F05/v of the second left pixel in FIG. 18, with thefirst shutter time/v since the time of shutter opening, is equal to thethird foreground component of the fourth left pixel in FIG. 18corresponding to the second shutter time/v since the time of shutteropening. Similarly, the foreground component F05/v is equal to theforeground component of the fourth left pixel in FIG. 18 correspondingto the third shutter time/v since the time of shutter opening and to theforeground component of the fifth left pixel in FIG. 18 corresponding tothe fourth shutter time/v since the time of shutter opening.

Since the object corresponding to the foreground is a rigid body, suchthat the foreground picture is moved at an equal speed so as to bedisplayed four pixels towards right in the next frame, the firstforeground component F04/v of the leftmost pixel in FIG. 18, with thefirst shutter time/v since the time of shutter opening, is equal to thesecond foreground component of the second left pixel in FIG. 18corresponding to the second shutter time/v since the time of shutteropening. Similarly, the foreground component F04/v is equal to theforeground component of the third left pixel in FIG. 18 corresponding tothe third shutter time/v since the time of shutter opening and to theforeground component of the fourth left pixel in FIG. 18 correspondingto the fourth shutter time/v since the time of shutter opening.

The foreground area corresponding to the moving object thus contains themotion blurring and hence may be said to be a distorted area.

FIG. 19 diagrammatically shows pixel values of pixels on a linecomprehending the uncovered background area in case the foreground ismoving towards right in the drawing, with the pixels shown extended inthe time axis direction. In FIG. 19, the movement quantity v of theforeground is 4. Since one frame is of short duration, it may be assumedthat the object corresponding to the foreground is a rigid body andmoving at an equal speed. In FIG. 19, the picture of the objectcorresponding to the foreground is moved so as to be displayed fourpixels rightwards in a frame next to a directly previous frame.

In FIG. 19, the leftmost to fourth left pixels belong to the backgroundarea. In FIG. 19, fifth left to seventh left pixels belong to the mixedarea which is the covered background area. In FIG. 19, the rightmostpixel belongs to the background area.

Since the object corresponding to the foreground which has hidden theobject corresponding to the background is moved so as to be removed froma position ahead of the object corresponding to the background, as timeelapses, the components contained in the pixel values of the pixelsbelonging to the covered background area are switched from thebackground component picture to the foreground component picture at acertain time point of the time period corresponding to the shutter time.

For example, the pixel value M′, shown with a bold line frame in FIG.18, is represented by the equation (12):M′=F02/v+F01/v+B26/v+B26/v  (12).

For example, the fifth left pixel contains the background componentcorresponding to three shutter time/v and the foreground componentcorresponding to one shutter time/v, the mixing ratio α of the fifthleft pixel is ¾. The sixth left pixel contains the background componentcorresponding to two shutter time/v and the foreground componentcorresponding to two shutter time/v, so the mixing ratio α is ½. Theseventh left pixel contains the background component corresponding toone shutter time/v and the foreground component corresponding to threeshutter time/v, so the mixing ratio α is ¼.

56.

If the equations (11), (12) are generalized, the pixel value M isrepresented by the following equation (13):

$\begin{matrix}{M = {{\alpha \cdot \beta} + {\sum\limits_{i}\;{{Fi}/v}}}} & (13)\end{matrix}$where α is a mixing ratio, B is a pixel value of the background and Fi/vis the foreground component.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at an equal speed, with the movement quantity vbeing 4, the first foreground component F01/v of the fifth left pixel inFIG. 19, with the first shutter time/v since the time of shutteropening, is equal to the second foreground component of the sixth leftpixel in FIG. 19 corresponding to the second shutter time/v since thetime of shutter opening. Similarly, the F01/v is equal to the foregroundcomponent of the seventh left pixel in FIG. 19 corresponding to thethird shutter time/v since the time of shutter opening and to theforeground component of the eighth left pixel in FIG. 19 correspondingto the fourth shutter time/v since the time of shutter opening.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at an equal speed, with the movement quantity vbeing 4, the first foreground component F02/v of the sixth left pixel inFIG. 19, with the first shutter time/v since the time of shutteropening, is equal to the second foreground component of the seventh leftpixel in FIG. 19 corresponding to the second shutter time/v since thetime of shutter opening. Similarly, the foreground component F02/v isequal to the foreground component of the eighth left pixel in FIG. 19corresponding to the third shutter time/v since the time of shutteropening.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at an equal speed, with the movement quantity vbeing 4, the first foreground component F03/v of the seventh left pixelin. FIG. 19, with the first shutter time/v since the time of shutteropening, is equal to the second foreground component of the eighth leftpixel in FIG. 19 corresponding to the second shutter time/v since thetime of shutter opening.

Although the number of times of the virtual splitting is four in thedescription with respect to FIGS. 17 to 19, the number of times of thevirtual splitting corresponds to the movement quantity v. The movementquantity v generally corresponds to the movement speed of the objectcorresponding to the foreground. For example, if the objectcorresponding to the foreground is moving so as to be displayed fourpixels rightwards in a frame next to a previous reference frame, themovement quantity v is 4. The number of times of the virtual splittingis set to 4 in association with the movement quantity v. Similarly, ifthe object corresponding to the foreground is moving so as to bedisplayed six pixels rightwards in a frame next to a previous referenceframe, the movement quantity v is 6, with the number of times of thevirtual splitting being six.

FIGS. 20 and 21 show the relation between the foreground area,background area, and the mixed area, comprised of the covered backgroundarea and the uncovered background area, on one hand, and the foregroundand background components corresponding to the split shutter time, onthe other hand, as described above.

FIG. 20 shows an example of extraction of pixels of the foreground area,background area and the mixed area as extracted from a picturecorresponding to an object moving before a still background. In theembodiment shown in FIG. 20, an object corresponding to the foregroundis moving horizontally with respect to the picture.

The frame #n+1 is a frame next to the frame #n, with the frame #n+2being a frame next to the frame #n+1.

FIG. 21 diagrammatically shows a model obtained on extracting pixels ofthe foreground area, background area and the mixed area, extracted inturn from one of the frames #n to #n+2, with the movement quantity vbeing 4, and on expanding the pixel values of the extracted pixels alongthe time axis direction.

Since the object corresponding to the foreground is moved, the pixelvalues of the foreground area are constituted by four differentforeground components corresponding to the period of the shutter time/v.For example, the leftmost one of pixels of the foreground area shown inFIG. 21 are F01/v, F02/v, F03/v and F04/v. That is the pixels of theforeground area contain are corrupted with motion blurring.

Since the object corresponding to the background is at a standstill, thelight corresponding to the background input to the sensor 11 during thetime corresponding to the shutter time is not changed. In this case, thepixel values of the background are free of the motion blurring.

The pixel values or the pixels belonging to the mixed area composed ofthe covered background area or the uncovered background area arecomprised of the foreground and background components.

A model comprised of neighboring pixels in a row in plural frames, inwhich the pixel values of pixels lying at the same position on a frameare developed in the time axis direction, with the picture correspondingto an object being moved, is explained. For example, if the picturecorresponding to the object is moving horizontally with respect to thepicture, the pixels arrayed on the same row on the picture may beselected as the pixels in a row in a picture.

FIG. 22 diagrammatically shows a model obtained on temporally expandingthe pixel values of pixels arrayed in a row of each of three frames of apicture of a photographed object corresponding to a still background,with the developed pixels being at the same positions on the respectiveframes. The frame #n is the frame next to the frame #n−1, with the frame#n+1 being the frame next to the frame #n. The remaining frames aretermed in similar manner.

The pixel values of B01 to B12 shown in FIG. 22 are those of pixelscorresponding to the object of the still background. Since the objectcorresponding to the background is at a standstill, the pixel values ofthe corresponding pixels in the frames #n·1 ti frame n+1 are notchanged. For example, the pixel in the frame #n and the pixel in theframe #n+1, corresponding to the positions of the pixels having pixelvalues of B05 in the frame #n·1, are of pixel values of B05.

FIG. 23 shows pixel values of neighboring pixels in a row in each ofthree frames of a photographed picture of an object corresponding to theforeground moving rightwards in FIG. 23, along with the objectcorresponding to the still background, with the pixel values being showndeveloped along the time axis direction. The model shown in FIG. 23includes a covered background area.

In FIG. 23, the object corresponding to the foreground is a rigid bodyand may be assumed to be moving at a constant speed, with the foregroundpicture being moved so that the foreground picture will be displayedfour pixels rightwards in the next frame. So, the movement quantity v ofthe foreground is 4, with the number of times of the virtual splittingbeing 4.

For example, the foreground component of the leftmost pixel of the frame#n·1 in FIG. 23, with the first shutter time/v since the opening of theshutter, is F12/v, whilst the foreground component of the second leftpixel, with the second shutter time/v since the opening of the shutter,is also F12v. The foreground component of the third left pixel in FIG.23, with the third shutter time/v since the opening of the shutter, andthe foreground component of the fourth left pixel in FIG. 23, with thefourth shutter time/v since the opening of the shutter, are each F12/v.

For example, the foreground component of the leftmost pixel of the frame#n·1 in FIG. 23, with the second shutter time/v since the opening of theshutter, is F11/v, whilst the foreground component of the second leftpixel, with the third shutter time/v since the opening of the shutter,is also F11v. The foreground component of the third left pixel in FIG.23, with the fourth shutter time/v since the opening of the shutter, isF11/v.

The foreground component of the leftmost pixel of the frame #n·1 in FIG.23, with the third shutter time/v since the opening of the shutter, isF10/v, whilst the foreground component of the second left pixel, withthe fourth shutter time/v since the opening of the shutter, is alsoF10v. The foreground component of the leftmost pixel in FIG. 23, withthe fourth shutter time/v since the opening of the shutter, is F09/v.

Since the object corresponding to the background is at a standstill, thebackground component of the second left pixel of the frame #n·1 in FIG.23, with the first shutter time/v as from the shutter opening time, isB01/v. The background component of the third left pixel of the frame#n·1 in FIG. 23, with the first and second shutter time/v as from theshutter opening time, is B02/v, while the background component of thefourth left pixel of the frame #n·1 in FIG. 23, with the first to thirdshutter time/v as from the shutter opening time, is B03/v.

In the frame #n·1 in FIG. 23, the leftmost pixel belongs to theforeground area, while the second to fourth left pixels belong to themixed area which is the covered background area.

The fifth to twelfth left pixels of the frame #n·1 in FIG. 23 belong tothe background area, with the corresponding pixel values being B04 toB11, respectively.

The first to fifth pixels of the frame #n·1 in FIG. 23 belong to thebackground area. The foreground component in the foreground area of theframe #n, with the shutter time/v, is one of F05v to F12/v.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at a constant speed, with the foreground picturebeing moved so that the foreground picture will be displayed four pixelsrightwards in the next frame, the foreground component of the fifth leftpixel of the frame #n· in FIG. 23, with the first shutter time/v sincethe opening of the shutter, is F12/v, whilst the foreground component ofthe sixth left pixel, with the second shutter time/v since the openingof the shutter, is also F12v. The foreground component of the seventhleft pixel in FIG. 23, with the third shutter time/v since the openingof the shutter, and the foreground component of the eighth left pixel inFIG. 23, with the fourth shutter time/v since the opening of theshutter, are each F12/v.

The foreground component of the fifth left pixel of the frame #n in FIG.23, with the second shutter time/v since the opening of the shutter, isF11/v, whilst the foreground component of the sixth left pixel, with thethird shutter time/v since the opening of the shutter, is also F11v. Theforeground component of the seventh left pixel in FIG. 23, with thefourth shutter time/v since the opening of the shutter, is F11/v.

The foreground component of the fifth left pixel of the frame #n in FIG.23, with the third shutter time/v since the opening of the shutter, isF10/v, whilst the foreground component of the sixth left pixel, with thefourth shutter time/v since the opening of the shutter, is also F10v.The foreground component of the fifth left pixel in FIG. 23, with thefourth shutter time/v since the opening of the shutter, is F09/v.

Since the object corresponding to the background is at a standstill, thebackground component of the sixth left pixel of the frame #n in FIG. 23,with the first shutter time/v as from the shutter opening time, isB05/v. The background component of the seventh left pixel of the frame#n in FIG. 23, with the first and second shutter time/v as from theshutter opening time, is B06/v, while the background component of theeighth left pixel of the frame #n in FIG. 23, with the first to thirdshutter time/v as from the shutter opening time, is B07/v.

In the frame #n·1. in FIG. 23, the first to ninth left pixels belong tothe foreground area, while the sixth to eighth left pixels belong to themixed area which is the covered background area.

The first to ninth to twelfth left pixels of the frame #n+1 in FIG. 23belong to the foreground area, with the pixel values being B08 to B11,respectively.

The first to ninth pixels of the frame #n+1 in FIG. 23 belong to theforeground area. The foreground component in the foreground area of theframe #n+1, with the shutter time/v, is one of F01v to F12/v.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at a constant speed, with the foreground picturebeing moved so that the foreground picture will be displayed four pixelsrightwards in the next frame, the foreground component of the ninth leftpixel of the frame #n+1 in FIG. 23, with the first shutter time/v sincethe opening of the shutter, is F12/v, whilst the foreground component ofthe tenth left pixel, with the second shutter time/v since the openingof the shutter, is also F12v. The foreground component of the eleventhleft pixel in FIG. 23, with the third shutter time/v since the openingof the shutter, and the foreground component of the twelfth left pixelin FIG. 23, with the fourth shutter time/v since the opening of theshutter, are each F12/v.

The foreground component of the ninth left pixel of the frame #n+1 inFIG. 23, with the second shutter time/v since the opening of theshutter, is F11/v, whilst the foreground component of the tenth leftpixel, with the third shutter time/v since the opening of the shutter,is also F11v. The foreground component of the eleventh left pixel inFIG. 23, with the fourth shutter time/v since the opening of theshutter, is F11/v.

The foreground component of the ninth left pixel of the frame #n+1 inFIG. 23, with the third shutter time/v since the opening of the shutter,is F10/v, whilst the foreground component of the tenth left pixel, withthe fourth shutter time/v since the opening of the shutter, is alsoF10v. The foreground component of the ninth left pixel of the frame #n+1in FIG. 23, with the fourth shutter time/v since the opening of theshutter, is F09/v.

Since the object corresponding to the background is at a standstill, thebackground component of the tenth left pixel of the frame #n+1 in FIG.23, with the first shutter time/v as from the shutter opening time, isB09/v. The background component of the eleventh left pixel of the frame#n+1 in FIG. 23, with the first and second shutter time/v as from theshutter opening time, is B10/v, while the background component of thetwelfth left pixel of the frame #n+1 in FIG. 23, with the first to thirdshutter time/v as from the shutter opening time, is B11/v.

In the frame #n+1 in FIG. 23, the tenth to twelfth left pixelscorrespond to the mixed area which is the covered background area.

FIG. 24 diagrammatically shows a picture obtained on extracting theforeground component from the pixel values shown in FIG. 23.

FIG. 25 shows neighboring pixels in a row of each of three frames of aphotographed picture of the foreground corresponding to an object movingrightwards in the drawing, along with the still background. In FIG. 25,there is also shown the uncovered background area.

In FIG. 25, the object corresponding to the foreground is a rigid bodyand may be assumed to be moving at a constant speed, with the foregroundpicture being moved so that the foreground picture will be displayedfour pixels rightwards in the next frame. So, the movement quantity v ofthe foreground is 4.

For example, the foreground component of the leftmost pixel of the frame#n·1 in FIG. 25, with the first shutter time/v since the opening of theshutter, is F13/v, whilst the foreground component of the second leftpixel, with the second shutter time/v since the opening of the shutter,is also F13v. The foreground component of the third left pixel in FIG.23, with the second shutter time/v since the opening of the shutter, andthe foreground component of the fourth left pixel in FIG. 25, with thefourth shutter time/v since the opening of the shutter, are each F13/v.

For example, the foreground component of the second left pixel of theframe #n·1 in FIG. 23, with the first shutter time/v since the openingof the shutter, is F14/v, whilst the foreground component of the thirdleft pixel, with the second shutter time/v since the opening of theshutter, is also F14v. The foreground component of the third left pixelin FIG. 25, with the first shutter time/v since the opening of theshutter, is F15/v.

Since the object corresponding to the background is at a standstill, thebackground component of the leftmost pixel of the frame #n·1 in FIG. 25,with the second to fourth shutter time/v as from the shutter openingtime, is B01/v. The background component of the second left pixel of theframe #n·1 in FIG. 25, with the third and fourth shutter time/v as fromthe shutter opening time, is B26/v, while the background component ofthe third left pixel of the frame #n·1 in FIG. 25, with the fourthshutter time/v as from the shutter opening time, is B27/v.

In the frame #n·1 in FIG. 25, the first to third left pixel belongs tothe mixed area which is the covered background area.

The fourth to twelfth left pixels of the frame #n·1 in FIG. 25 belong tothe foreground area, with the foreground component of the foreground ofthe frame being one of F13v to F24v.

The first to fourth left pixels of the frame #n in FIG. 25 belong to thebackground area, with the pixel values being B25 to B28, respectively.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at a constant speed, with the foreground picturebeing moved so that the foreground picture will be displayed four pixelsrightwards in the next frame, the foreground component of the fifth leftpixel of the frame #n in FIG. 23, with the first shutter time/v sincethe opening of the shutter, is F13/v, whilst the foreground component ofthe sixth left pixel, with the second shutter time/v since the openingof the shutter, is also F13v. The foreground component of the seventhleft pixel in FIG. 25, with the third shutter time/v since the openingof the shutter, and the foreground component of the eighth left pixel inFIG. 25, with the fourth shutter time/v since the opening of theshutter, are each F13/v.

The foreground component of the sixth left pixel of the frame #n in FIG.23, with the first shutter time/v since the opening of the shutter, isF14/v, whilst the foreground component of the seventh left pixel, withthe second shutter time/v since the opening of the shutter, is alsoF14v. The foreground component of the eighth left pixel in FIG. 25, withthe first shutter time/v since the opening of the shutter, is F15/v.

Since the object corresponding to the background is at a standstill, thebackground component of the fifth left pixel of the frame #n in FIG. 25,with the second to fourth shutter time/v as from the shutter openingtime, is B29/v. The background component of the sixth left pixel of theframe #n in FIG. 25, with the third and fourth shutter time/v as fromthe shutter opening time, is B30/v, while the background component ofthe seventh left pixel of the frame #n in FIG. 23, with the fourthshutter time/v as from the shutter opening time, is B31/v.

In the frame #n in FIG. 25, the first to ninth left pixels belong to theforeground area, while the fifth to seventh left pixels belong to themixed area which is the covered background area.

The eighth to twelfth left pixels of the frame #n+1 in FIG. 25 belong tothe foreground area, with the pixel values being B25 to B32,respectively.

The first to eighth pixels of the frame #n+1 in FIG. 25 belong to thebackground area, with the pixel values being B25 to B32, respectively.

Since the object corresponding to the foreground is a rigid body and maybe assumed to be moving at a constant speed, with the foreground picturebeing moved so that the foreground picture will be displayed four pixelsrightwards in the next frame, the foreground component of the ninth leftpixel of the frame #n+1 in FIG. 25, with the first shutter time/v sincethe opening of the shutter, is F13/v, whilst the foreground component ofthe tenth left pixel, with the second shutter time/v since the openingof the shutter, is also F13v. The foreground component of the eleventhleft pixel in FIG. 25, with the third shutter time/v since the openingof the shutter, and the foreground component of the twelfth left pixelin FIG. 25, with the fourth shutter time/v since the opening of theshutter, are each F13/v.

The foreground component of the tenth left pixel of the frame #n+1 inFIG. 25, with the first shutter time/v since the opening of the shutter,is F14/v, whilst the foreground component of the eleventh left pixel,with the second shutter time/v since the opening of the shutter, is alsoF14v. The foreground component of the twelfth left pixel in FIG. 25,with the first shutter time/v since the opening of the shutter, isF15/v.

Since the object corresponding to the background is at a standstill, thebackground component of the ninth left pixel of the frame #n+1 in FIG.25, with the second to fourth shutter time/v as from the shutter openingtime, is B33/v. The background component of the tenth left pixel of theframe #n+1 in FIG. 25, with the third and fourth shutter time/v as fromthe shutter opening time, is B34/v, while the background component ofthe eleventh left pixel of the frame #n+1 in FIG. 25, with the fourthshutter time/v as from the shutter opening time, is B35/v.

In the frame #n+1 in FIG. 25, the ninth to eleventh left pixelscorrespond to the mixed area which is the covered background area.

In FIG. 25, the twelfth left pixel of the frame #n+1 belong to theforeground area. The foreground component with the shutter time/v in theforeground area of frame #n+1 is one of F13v to F16v.

FIG. 26 diagrammatically shows a picture obtained on extracting theforeground component from the pixel values shown in FIG. 25.

Reverting to FIG. 10, the area specifying unit 103 associates a flag,indicating that a given picture belong to the foreground area, abackground area, a covered background area or an uncovered backgroundarea, from pixel to pixel, using pixel value of plural frames, androutes the resulting areal information to the mixing ratio calculatingunit 104 and to the motion blurring adjustment unit 106.

Based on the pixel values of plural frames and the areal information,the mixing ratio calculating unit 104 computes the mixing ratio α foreach of the pixels contained in the mixed area, and sends the computedmixing ratio α to the foreground/background separating unit 105.

Based on the pixel values of the plural frames, areal information andthe mixing ratio α, the foreground/background separating unit 105extracts the foreground component picture made up only of the foregroundcomponent to send the extracted component picture to the motion blurringadjustment unit 106.

Based on the foreground component picture sent from theforeground/background separating unit 105, the motion vector sent fromthe motion detection unit 102 and on the areal information sent from thearea specifying unit 103, the motion blurring adjustment unit 106adjusts the quantity of the motion blurring contained in the foregroundcomponent picture to output the foreground component picture adjustedfor the motion blurring.

Referring to the flowchart of FIG. 27, the processing for adjusting themotion blurring caused by the signal processor 12 is explained. At stepS101, the area specifying unit 103 executes the area specifyingprocessing for generating the areal information indicating to which ofthe foreground area, background area, covered background area or theuncovered background area belong the pixels of the input picture, fromone pixel of the input picture to another. The area specifyingprocessing will be explained subsequently by referring to the flowchartof FIG. 36. The area specifying unit 103 sends the generated areainformation to the mixing ratio calculating unit 104.

Meanwhile, the area specifying unit 103 at step S101 may generate theareal information indicating to which of the foreground area, backgroundarea or the mixed area belong the pixels of the input picture, from onepixel of the input picture to another, based on the input picture. Inthis case, no distinction is made between the covered background areaand the uncovered background area. In this case, theforeground/background separating unit 105 and the motion blurringadjustment unit 106 decide whether the mixed area is the coveredbackground area or the uncovered background area, based on the directionof the motion vector. For example, if the foreground area, mixed areaand the background area are arrayed sequentially in association with thedirection of the motion vector, the mixed area is verified to be thecovered background area, whereas, if the background area, mixed area andthe foreground area are arrayed sequentially in association with thedirection of the motion vector, the mixed area is verified to be theuncovered background area.

At step S102, the mixing ratio calculating unit 104 calculates themixing ratio α, from one pixel contained in the mixing area to another,based on the input picture and the area information. The processing forcomputing the mixing ratio will be explained in detail subsequently byreferring to the flowchart of FIG. 46. The mixing ratio calculating unit104 sends the computed mixing ratio α to the foreground/backgroundseparating unit 105.

At step S103, the foreground/background separating unit 105 extracts theforeground component from the input picture, based on the motion vectorand the areal information, to send the extracted component to the motionblurring adjustment unit 106 as the foreground component picture.

At step S104, the motion blurring adjustment unit 106 generates aprocessing unit for indicating a position on the picture of pixelsarrayed consecutively in the movement direction of each of the uncoveredbackground area, foreground area and the covered background area, basedon the motion vector and on the area information, to adjust the quantityof the motion blurring contained in the foreground componentcorresponding to the processing unit. The processing for adjusting thequality of the motion blurring will be explained subsequently byreferring to the flowchart of FIG. 63.

At step S105, the signal processor 12 verifies whether or not theprocessing has been finished for the entire picture. If the signalprocessor 12 has verified that the processing has not been finished fortthe entire picture, it proceeds to step S104 to repeat the processingfor adjusting the quantity of the motion blurring for the foregroundcomponent corresponding to the processing unit.

If, at step S106, it is verified that the processing has been finishedfor the entire picture, the processing is terminated.

In this manner, the signal processor 12 is able to separate theforeground and the background from each other to adjust the quantity ofthe motion blurring contained in the foreground. That is, the signalprocessor 12 is able to adjust the amount of motion blurring containedin sample data as pixel value of the foreground pixel.

In the following, illustrative structures of the area specifying unit103, mixing ratio calculating unit 104, foreground/background separatingunit 105 and the motion blurring adjustment unit 106 are hereinafterexplained.

FIG. 28 is a block diagram showing an illustrative structure of the areaspecifying unit 103. A frame memory 121 stores an input picture on theframe basis. When a frame being processed is a frame #n, the framememory 121 stores a frame #n·2, as a frame two frames before the frame#n, a frame #n·1, as a frame one frame before the frame #n, a frame#n+1, as a frame one frame after the frame #n, and a frame #n+2, as aframe two frames after the frame #n.

A still/movement discriminating unit 122-1 reads out a pixel value of apixel of the frame #n+2 lying at the same position as the position onthe picture of the pixel of the frame #n being area-specified, and apixel value of a pixel of the frame #n+1 lying at the same position asthe position on the picture of the pixel of the frame #n beingarea-specified, from the frame memory 121, to calculate an absolutevalue of the difference of the read-our pixel values. The still/movementdiscriminating unit 122-1 verifies whether or not the absolute value ofthe difference between the pixel value of the frame #n+2 and the frame#n+1 is larger than a predetermined threshold value Th. If it isverified that the absolute value of the difference is larger than thethreshold value Th, the still/movement discriminating unit 122-1 routesa still/movement decision specifying the movement decision to an areadecision unit 123-1. If it is verified that the absolute value o thedifference between the pixel value of the frame #n+2 nd the pixel valueof the frame #n+1 is not larger than the threshold value Th, thestill/movement discriminating unit 122-1 routes a still/movementdecision specifying the still decision to an area decision unit 123-1.

A still/movement discriminating unit 122-2 reads out a pixel value of apixel of the frame #n+1 lying at the same position as the position onthe picture of the pixel of the frame #n being area-specified, and apixel value of a pixel of the frame #n+1 lying at the same position asthe position on the picture of the pixel of the frame #n beingarea-specified, from the frame memory 121, to calculate an absolutevalue of the difference of the read-our pixel values. The still/movementdiscriminating unit 122-2 verifies whether or not the absolute value ofthe difference between the pixel value of the frame #n+1 and the frame#n is larger than a predetermined threshold value Th. If it is verifiedthat the absolute value of the difference between the is larger than thethreshold value Th, the still/movement discriminating unit 122-1 routesa still/movement decision specifying the movement decision to an areadecision unit 123-land to an area decision unit 123-2. If it is verifiedthat the absolute value of the difference between the pixel value of thepixel of the frame #n+1 and that of the pixel of the frame #n is notlarger than the threshold value Th, the still/movement discriminatingunit 122-1 routes a still/movement decision specifying the stilldecision to an area decision unit 123-1 and to an area decision unit123-2.

A still/movement discriminating unit 122-3 reads out a pixel value of apixel of the frame #n lying at the same position as the position on thepicture of the pixel of the frame #n being area-specified, and a pixelvalue of a pixel of the frame #n·1 lying at the same position as theposition on the picture of the pixel of the frame #n beingarea-specified, from the frame memory 121, to calculate an absolutevalue of the difference of the read-our pixel values. The still/movementdiscriminating unit 122-3 verifies whether or not the absolute value ofthe difference between the pixel value of the frame #n and the frame#n·1 is larger than a predetermined threshold value Th. If it isverified that the absolute value of the difference between the pixelvalues is larger than the threshold value Th, the still/movementdiscriminating unit 122-3 routes a still/movement decision specifyingthe movement decision to an area decision unit 123-1 and to an areadecision unit 123-3. If it is verified that the absolute value of thedifference between the pixel value of the pixel of the frame #n and thatof the pixel of the frame #n·1 is not larger than the threshold valueTh, the still/movement discriminating unit 122-3 routes a still/movementdecision specifying the still decision to an area decision unit 123-2and to an area decision unit 123-3.

A still/movement discriminating unit 122-4 reads out the pixel value ofthe pixel of the frame #n·1 lying at the same position as the positionon the picture of the pixel of the frame #·n being area-specified, andthe pixel value of the pixel of the frame #n·2 lying at the sameposition on the picture of the pixel of the frame #n beingarea-specified, to calculate the absolute value of the difference of thepixel values. The still/movement discriminating unit 122-4 verifieswhether or not the absolute value of the difference of the pixel valueof the frame #n·1 and the pixel value of the frame #n·2 is larger thanthe predetermined threshold value Th. If the absolute value of thedifference between the pixel value of the frame #n·1 and the pixel valueof the frame #n·2 is verified to be larger than the threshold value Th,a still/movement decision indicating the decision for movement is routedto the area decision unit 123-3. If it is verified that the absolutevalue of the difference between the pixel value of the frame #n·1 andthe pixel value of the frame #n·2 is not larger than the threshold valueTh, the still/movement discriminating unit 122-4 routes a still/movementdecision indicating the still decision to the area decision unit 123-3.

If the still/movement decision routed from the still/movementdiscriminating unit 122-1 indicates still and the still/movementdecision routed from the still/movement discriminating unit 122-2indicates movement, the area decision unit 123-1 decides that the pixelon the frame #n being area-specified belongs to the uncovered backgroundarea and sets “1” in an uncovered background area decision flagassociated with the pixel being area-specified for indicating that thepixel belongs to the uncovered background area.

If the still/movement decision routed from the still/movementdiscriminating unit 122-1 indicates movement and the still/movementdecision routed from the still/movement discriminating unit 122-2indicates still, the area decision unit 123-1 decides that the pixel onthe frame #n being area-specified does not belong to the uncoveredbackground area and sets “0” in an uncovered background area decisionflag associated with the pixel being area-specified for indicating thatthe pixel does not belong to the uncovered background area.

The area decision unit 123-1 routes the uncovered background areadecision flag, having “1” or “0” set in this manner, to a decision flagstorage memory 124.

If the still/movement decision routed from the still/movementdiscriminating unit 122-2 indicates still and the still/movementdecision routed from the still/movement discriminating unit 122-3indicates still, the area decision unit 123-2 decides that the pixel onthe frame #n being area-specified belongs to the still area and sets “1”in a still area decision flag associated with the pixel beingarea-specified for indicating that the pixel belongs to the uncoveredbackground area.

If the still/movement decision routed from the still/movementdiscriminating unit 122-2 indicates movement or the still/movementdecision routed from the still/movement discriminating unit 122-3indicates movement, the area decision unit 123-2 decides that the pixelon the frame #n being area-specified does not belong to the still areaand sets “0” in a still area decision flag associated with the pixelbeing area-specified for indicating that the pixel does not belong tothe still area.

The area decision unit 123-2 routes the still area decision flag, thushaving “1” or “0” set therein, to the decision flag storage memory 124.

If the still/movement decision routed from the still/movementdiscriminating unit 122-2 indicates movement and the still/movementdecision routed from the still/movement discriminating unit 122-3indicates movement, the area decision unit 123-2 decides that the pixelon the frame #n being area-specified belongs to the movement area andsets “1” in a movement area decision flag associated with the pixelbeing area-specified for indicating that the pixel belongs to themovement area.

If the still/movement decision routed from the still/movementdiscriminating unit 122-2 indicates still or the still/movement decisionrouted from the still/movement discriminating unit 122-3 indicatesstill, the area decision unit 123-2 decides that the pixel on the frame#n being area-specified does not belong to the movement area and sets“0” in a movement area decision flag associated with the pixel beingarea-specified for indicating that the pixel does not belong to themovement area.

The area decision unit 123-2 routes the movement area decision flag,thus having “1” or “0” set therein, to the decision flag storage memory124.

If the still/movement decision routed from the still/movementdiscriminating unit 122-3 indicates movement and the still/movementdecision routed from the still/movement discriminating unit 122-4indicates still, the area decision unit 123-3 decides that the pixel onthe frame #n being area-specified belongs to the uncovered backgroundarea and sets “1” in a covered background area decision flag associatedwith the pixel being area-specified for indicating that the pixelbelongs to the covered background area.

If the still/movement decision routed from the still/movementdiscriminating unit 122-3 indicates still or the still/movement decisionrouted from the still/movement discriminating unit 122-4 indicatesmovement, the area decision unit 123-3 decides that the pixel on theframe #n being area-specified does not belong to the covered backgroundarea and sets “0” in a covered background area decision flag associatedwith the pixel being area-specified for indicating that the pixel doesnot belong to the covered background area.

The area decision unit 123-3 routes the covered background area decisionflag, thus having “1” or “0” set therein, to the covered background areadecision flag storage memory 124.

The decision flag storage memory 124 stores the uncovered backgroundarea decision flag, sent from the area decision unit 123-1, the stillarea decision flag, sent from the area decision unit 123-2, the movementarea decision flag, sent from the area decision unit 123-2, and theuncovered background area decision flag, sent from the area decisionunit 123-3.

The decision flag storage memory 124 sends the uncovered background areadecision flag, still area decision flag, movement area decision flag andthe covered background area decision flag to a synthesis unit 125. Basedon the uncovered background area decision flag, still area decisionflag, movement area decision flag and the covered background areadecision flag, supplied from the decision flag storage memory 124, thesynthesis unit generates the area information indicating to which of theuncovered background area, still area, movement area and the coveredbackground area belong the respective pixels, and routes the informationso generated to a decision flag storage frame memory 126.

The decision flag storage frame memory 126 stores the area information,supplied from the synthesis unit 125, while outputting the areainformation stored therein.

Referring to FIGS. 29 to 33, a typical processing by the area specifyingunit 103 is explained.

When an object corresponding to the foreground is moving, the positionof the picture corresponding to the object on the picture screen ischanged from frame to frame. Referring to FIG. 29, a picturecorresponding to an object at a position Yn(x, y) in a frame #n ispositioned at Yn+1(x,y) at the next frame #n+1.

FIG. 30 diagrammatically shows a model of pixel values of a row ofpixels neighboring to one another along the moving direction of thepicture corresponding to the foreground. For example, if the movementdirection of the picture corresponding to the foreground is horizontalrelative to the picture screen, the diagrammatic view of FIG. 30 shows amodel in which pixel values of pixels neighboring to one another on oneline are developed in the time axis direction.

In FIG. 30, the line in the frame #n is the same as one in the frame#n+1.

The components of the foreground corresponding to the object containedin the second to the thirteenth pixels as counted from left in the frame#n are included in the sixth to seventeenth pixels as counted from theleft of the frame #n+1.

The pixels belonging to the covered background area in the frame #n arethe eleventh to thirteenth pixels as counted from left, whilst thepixels belonging to the uncovered background area are the second tofourth pixels as counted from left. The pixels belonging to the coveredbackground area in the frame #n+1 are the fifteenth to seventeenthpixels as counted from left, whilst the pixels belonging to theuncovered background area are the sixth to eighth pixels as counted fromleft.

In the example shown in FIG. 30, since the foreground component in theframe #n are moved by four pixels in the frame #n+1, the movementquantity v is 4. The number of times of the virtual splittingcorresponds to the movement quantity and is equal to 4.

The change in the pixel values of pixels belonging to the mixed areaahead and at back of the frame being considered is explained.

In the frame #n shown in FIG. 31, in which the background is still andthe movement quantity of the foreground v is 4, pixels belonging to thecovered background area are fifteenth to seventeenth pixels from left.Since the movement quantity v is 4, the fifteenth to seventeenth pixelsfrom left in the directly previous frame #n·1 contain only thebackground components and belong to the background. The fifteenth toseventeenth pixels from left in the further previous frame #n·2 containonly the background components and belong to the background area.

Since the object corresponding to the background is still, the pixelvalue of the fifteenth pixel from the left of the frame #n·1 is notchanged from the pixel value of the fifteenth pixel from the left of theframe #n·2. Similarly, the pixel value of the sixteenth pixel from theleft of the frame #n·1 is not changed from the pixel value of thesixteenth pixel from the left of the frame #n·2, whilst the pixel valueof the seventeenth pixel from the left of the frame #n·1 is not changedfrom the pixel value of the seventeenth pixel from the left of the frame#n·2.

That is, the pixels of the frame #n·1 and the frame #n·2 correspondingto the pixels belonging to the covered background area in the frame #nare comprised only of the background components and are not changed, sothat the absolute value of the difference is substantially 0. So, thestill/movement decision on the pixels of the frame #n·1 and frame #n·2corresponding to the mixed area in the frame #n is made as being stillby the still/moving discriminating unit 122-4.

Since the pixels belonging to the covered background area in the frame#n contain the foreground components, the corresponding pixel valuesdiffer from those in which the pixels are comprised only of backgroundcomponents in the frame #n·1. Therefore, the pixels belonging to themixed area in the frame #n and the corresponding pixels of the frame#n·1 are verified to be moving pixels by the still/moving discriminatingunit 122-3.

When fed with the result of still/movement decision indicating themovement from the still/moving discriminating unit 122-3 and with theresult of still/movement decision indicating the still from thestill/moving discriminating unit 122-4, the area decision unit 123-3decides that the pixel in question belongs to the covered backgroundarea.

The pixels contained in the uncovered background area in the frame #n inwhich the background is still and the movement quantity v of theforeground is 4 are second to fourth pixels as counted from left. Sincethe movement quantity v is 4, the second to fourth pixels from left inthe next frame #n+1 contain only the background components and belong tothe background area. In the second next frame #n+2, the second to fourthpixels from left contain only the background components and belong tothe background area.

Since the object corresponding to the background is still, the pixelvalue of the second pixel from: left of the frame #n+2 is not changedfrom the pixel value of the second pixel from left of the frame #n+1.Similarly, the pixel value of the second pixel from left of the frame#n+2 is not changed from the pixel value of the second pixel from leftof the frame #n+1, whilst the pixel value of the third pixel from leftof the frame #n+2 is not changed from the pixel value of the fourthpixel from left of the frame #n+1.

That is, the pixels of the frame #n+1 and frame #n+2 corresponding tothe pixels belonging to the uncovered background area in the frame #nare composed only of background components and are not changed in thepixel values. So, the absolute value of the difference is approximatelyzero. Therefore, the pixels of the frame #n+1 and frame #n+2corresponding to the pixels belonging to the mixed area in the frame #nare decided by the still/moving discriminating unit 122-1 to be stillpixels.

The pixels belonging to the uncovered background area in the frame #ncontain the foreground components and hence differ in pixel values fromthe pixels in the frame #n+1 composed only of the background components.So, the pixels belonging to the mixed area in the frame #n and those ofthe corresponding frame #n·1 are decided by the still/movingdiscriminating unit 122-2 to be moving pixels.

The area decision unit 123-1 is fed in this manner with the resultindicating movement from the still/moving discriminating unit 122-2. Iffed with the result indicating still from the still/movingdiscriminating unit 122-1, the area decision unit 123-1 decides that thecorresponding pixel belongs to the uncovered background area.

FIG. 33 shows decision conditions of the area specifying unit 103 in theframe #n. When the pixel of the frame #n·2 at the same position as theposition on the picture of the pixel of the frame #n being verified andthe pixel of the frame #n·1 at the same position as the position on thepicture of the pixel of the frame #n being verified, are decided to bestill, whilst the pixel of the frame #n·1 at the same position as theposition on the picture of the pixel of the frame #n being verified andthe pixel of the frame #n are decided to be moving, the area specifyingunit 103 decides that the pixel of the frame #n being verified belongsto the covered background area.

When the pixel of the frame #n·1 at the same position as the position onthe picture of the pixel of the frame #n being verified and the pixel ofthe frame #n are decided to be still, whilst the pixel of the frame #nand the pixel of the frame #n+1 at the same position as the position onthe picture of the pixel of the frame #n being verified are decided tobe still, the area specifying unit 103 decides that the pixel of theframe #n being verified belongs to the still area.

When the pixel of the frame #n·1 at the same position as the position onthe picture of the pixel of the frame #n being verified and the pixel ofthe frame #n are decided to be moving, whilst the pixel of the frame #nand the pixel of the frame #n+1 at the same position as the position onthe picture of the pixel of the frame #n being verified are decided tobe still, the area specifying unit 103 decides that the pixel of theframe #n being verified belongs to the moving area.

When the pixel of the frame #n and the pixel of the frame #n+1 at thesame position as the position on the picture of the pixel of the frame#n being verified are decided to be moving and when the pixel of theframe #n+1 at the same position as the position on the picture of thepixel of the frame #n being verified and the pixel of the frame #n+1 atthe same position as the position on the picture of the pixel of theframe #n being verified and the pixel of the frame #n+2 at the sameposition as the position on the picture of the pixel of the frame #nbeing verified are decided to be still, the area specifying unit 103decides that the pixel of the frame #n being verified belongs to theuncovered background area.

FIG. 34 shows an example of the area decision by the area specifyingunit 103. In FIG. 34A, a pixel decided to belong to the coveredbackground area is shown in white. In FIG. 34B, a pixel decided tobelong to the uncovered background area is shown in white.

In FIG. 34C, a pixel decided to belong to the moving area is shown inwhite. In FIG. 34D, a pixel decided to belong to the still area is shownin white.

FIG. 35 shows the area information representing the mixed area, amongthe area information output by the decision flag storage frame memory126, as picture. In FIG. 35, the pixel decided to belong to the coveredbackground area or the uncovered background area, that is to the mixedarea, is shown in white. The area information indicating the mixed area,output by the decision flag storage frame memory 126, indicates atextured portion surrounded by an untextured portion in the foregroundarea and the mixed area.

Referring to the flowchart of FIG. 36, the processing for areaidentification by the area specifying unit 103 is explained. At stepS121, the frame memory 121 acquires pictures of the frame #n·2 to frame#n+2, inclusive the frame #n.

At step S122, the still/moving discriminating unit 122-3 checks whetheror not the pixels at the same position of the frame #n·1 and the frame#n are still. If the pixels are decided to be still, the program movesto step S123 where the still/moving discriminating unit 122-2 checkswhether or not the pixels at the same position of the frame #n and theframe #n+1 are still.

If, at step S123, the pixels at the same position of the frame #n andthe pixel of the frame #n+1 are decided to be still, the program movesto step S124 where the area decision unit 123-2 sets “1” in the stillarea decision flag corresponding to the pixel of the area being verifiedfor indicating that the pixel belongs to the still area. The areadecision unit 123-2 sends the still area decision flag to the decisionflag storage memory 124. The program then moves to step S125.

If at step S122 the pixels at the same position of the frame #n·1 andthe frame #n are decided to be moving or if at step S123 the pixels atthe same position of the frame #n and the frame #n+1. are decided to bemoving, the pixel of the frame #n does not belong to the still area, sothe processing at step S124 is skipped and the program moves to stepS125.

At step S125, the still/moving discriminating unit 122-3 checks whetheror not the pixels at the same position of the frame #n·1 and the frame#n are moving. If the pixels are decided to be moving, the program movesto step S126 where the still/moving discriminating unit 122-2 decideswhether or not the pixels at the same position of the frame #n and theframe #n+1 are moving.

If, at step S126, the pixels at the same position of the frame #n andthe pixel of the frame #n+1 are decided to be moving, the program movesto step S127 where the area decision unit 123-2 sets “1” in the movingarea decision flag corresponding to the pixel of the area being verifiedfor indicating that the pixel belongs to the moving area. The areadecision unit 123-2 sends the moving area decision flag to the decisionflag storage memory 124. The program then moves to step S128.

If at step S125 the pixels at the same position of the frame #n·1 andthe frame #n are decided to be still or if at step S126 the pixels atthe same position of the frame #n and the frame #n+1 are decided to bestill, the pixel of the frame #n does not belong to the moving area, sothe processing at step S127 is skipped and the program moves to stepS128.

At step S128, the still/moving discriminating unit 122-4 checks whetheror not the pixels at the same position of the frame #n·2 and the frame#n·1 are still. If the pixels are decided to be still, the program movesto step S129 where the still/moving discriminating unit 122-3 decideswhether or not the pixels at the same position of the frame #n·1 and theframe #n are moving.

If, at step S129, the pixels at the same position of the frame #n·1 andthe pixel of the frame #n are decided to be moving, the program moves tostep S130 where the area decision unit 123-3 sets “1” in the coveredbackground area decision flag corresponding to the pixel of the areabeing verified for indicating that the pixel belongs to the coveredbackground area. The area decision unit 123-3 sends the coveredbackground area decision flag to the decision flag storage memory 124.The program then moves to step S131.

If at step S128 the pixels at the same position of the frame #n·2 andthe frame #n·2 are decided to be moving or if at step S129 the pixels atthe same position of the frame #n·1 and the frame #n are decided to bestill, the pixel of the frame #n does not belong to the coveredbackground area, so the processing at step S130 is skipped and theprogram moves to step S131.

At step S131, the still/moving discriminating unit 122-2 checks whetheror not the pixels at the same position of the frame #n and the frame#n+1 are still. If the pixels are decided to be moving, the programmoves to step S132 where the still/moving discriminating unit 122-1decides whether or not the pixels at the same position of the frame #n+1and the frame #n+2 are moving.

If, at step S132, the pixels at the same position of the frame #n+1 andthe pixel of the frame #n+2 are decided to be still, the program movesto step S133 where the area decision unit 123-1 sets “1” in theuncovered background area decision flag corresponding to the pixel ofthe area being verified for indicating that the pixel belongs to theuncovered background area. The area decision unit 123-1 sends theuncovered background area decision flag to the decision flag storagememory 124. The program then moves to step S134.

If at step S131 the pixels at the same position of the frame #n and theframe #n+1 are decided to be still or if at step S132 the pixels at thesame position of the frame #n+1 and the frame #n+2 are decided to bemoving, the pixel of the frame #n does not belong to the uncoveredbackground area, so the processing at step S133 is skipped and theprogram moves to step S134.

At step S134, the area specifying unit 103 checks whether or not thearea has been specified for the totality of the pixels of the frame #n.If it is decided that the area has not been specified for the totalityof the pixels of the frame #n, the program reverts to step S122 torepeat the area specifying processing for the remaining pixels.

If it is decided at step S134 that the area has been specified for thetotality of the pixels of the frame #n, the program moves to step S135where the synthesis unit 125 generates the area information indicatingthe mixed area based on the uncovered background area decision flag andthe covered background area decision flag, stored in the decision flagstorage memory 124, while also generating the area informationindicating to which of the uncovered background area, still area, movingarea and the uncovered background area belongs each pixel. The synthesisunit 125 sets the generated area information in the decision flagstorage frame memory 126 to finish the processing.

In this manner, the area specifying unit 103 is able to generate thearea information, for each of pixels comprehended in a frame, indicatingthat the pixel in question belongs to the movement area, still area,covered background area or to the uncovered background area.

It is also possible for the area specifying unit 103 to apply logicalsum to area information corresponding to the uncovered background areaand the covered background area to generate the area informationcomprising a flag indicating that a given pixel contained in the framebelongs to the movement area, still area or to the mixed area, for eachpixel contained in the frame.

If the object associated with the foreground includes a texture, thearea specifying unit 103 is able to specify the movement area moreaccurately.

The area specifying unit 103 is able to output the area informationindicating the movement area as the area information indicating theforeground area, or the area information indicating the still area asthe area information indicating the background area.

In the foregoing, it is assumed that the object corresponding to thebackground is still. However, the above-described area specifyingprocessing can be applied even if the picture associated with thebackground area contains motion. For example, if the picturecorresponding to the background area is moving uniformly, the areaspecifying unit 103 shifts the entire picture in association with themovement to perform the processing in the same way as when the objectcorresponding to the background is still. If the picture associated withthe background area contains different movements from one location toanother, the area specifying unit 103 selects the pixels correspondingto the movement to perform the above processing.

FIG. 37 shows a block diagram showing an illustrative structure of themixing ratio calculating unit 104. The estimated mixing ratio processor201 calculates the estimated mixing ratio, from one pixel to another, bycalculations corresponding to the model of the covered background area,based on the input picture, to route the calculated estimated mixingratio to a mixing ratio decision unit 203.

An estimated mixing ratio processing unit 202 calculates the estimatedmixing ratio, from pixel to pixel, by calculations corresponding to themodel of the uncovered background area, based on the input picture, toroute the calculated mixing ratio to the mixing ratio decision unit 203.

Since the object corresponding to the foreground may be assumed to bemoving at an equal speed within the shutter time, the mixing ratio α ofa pixel belonging to the mixed area has the following properties: Thatis, the mixing ratio α is changed linearly relative to changes in thepixel positions. If the changes of the pixel positions areone-dimensional, the changes in the mixing ratio α can be represented asa plane.

Since the one-frame period is short, it may be assumed that the objectcorresponding to the foreground is a rigid member and is moving at anequal speed.

Meanwhile, the tilt of the mixing ratio α is inversely proportionate tothe movement quantity v of the foreground within the shutter time.

FIG. 38 shows an example of an ideal mixing ratio α. The tilt 1 in themixing area with an ideal mixing ratio α can be represented as areciprocal of the movement quantity v.

In the embodiment of FIG. 39, the pixel value C06 of the seventh pixelfrom left of the frame #n can be represented, using the pixel value P06of the seventh pixel from left of the frame #n·1, by the equation (14):

$\begin{matrix}\begin{matrix}{{C\; 06} = {{B\;{06/v}} + {B\;{06/v}} + {F\; 0\;{1/v}} + {F\;{02/v}}}} \\{= {{P\;{06/v}} + {P\;{06/v}} + {F\;{01/v}} + {F\;{02/v}}}} \\{= {{{{2/v} \cdot P}\; 06} + {\sum\limits_{i = 1}^{2}{F_{i}/v}}}}\end{matrix} & (14)\end{matrix}$

In the equation (14), the pixel value C06 is expressed as a pixel valueM of the pixel of the mixed area, whilst the pixel value P06 isexpressed as a pixel value B of the pixel of the background area. Thatis, the pixel value M of the mixed area and the pixel value B of thebackground may be represented by the equations (15) and (16),respectively:M=C06  (15)C=α·P+f  (16).

In the equation (14), 2/v corresponds to the mixing ratio α. Since themovement quantity v is 4, the mixing ratio α of the seventh pixel fromleft of the frame #n is 0.5.

By assuming that the pixel value C of the frame #n under considerationand the pixel value P of the frame #n·1 directly previous to the frame#n as being the pixel value of the mixed area and the pixel value of thebackground, respectively, the equation (13) indicating the mixing ratioα can be rewritten to the following equation (17):C=α·P+f  (17)where f denotes the sum Σ_(i)Fi/v of the foreground components containedin the considered pixel. There are two variables in the equation (17),namely the mixing ratio α and the sum f of the foreground components.

FIG. 40 shows a model in which the movement quantity v in the uncoveredbackground area is 4 and the number of times of the virtual splittingalong the time axis is 4, with the pixels being shown developed alongthe time axis direction.

By assuming, in the uncovered background area, that the pixel value C ofthe frame #n under consideration and the pixel value P of the frame #n+1next to the frame #n as being the pixel value of the mixed area and thepixel value of the background, respectively, as in the coveredbackground area, discussed above, the equation (13) indicating themixing ratio α can be represented as in the following equation (18):C=α·N+f  (18).

Although the background object is assumed to be still in the foregoingdescription, the equations (14) to (18) may be applied by exploiting thepixel values of the pixels associated with the background movementquantity v even if the background object is moving. For example, if,when the movement quantity v of the object corresponding to thebackground is 2 and the number of times of the virtual splitting is 2,the object corresponding to the background is moving towards right inthe drawing, the pixel value B of the pixel of the background area inthe equation (16) is the pixel value P04.

Since the equation (17) and (18) each contain two variables, the mixingarea α cannot be found directly. It should be noted that, since thepicture in general exhibits strong spatial correlation, the pixelsproximate to each other are of approximately the same pixel values.

Since the foreground components exhibits strong spatial correlation, theequation is modified so that the mixing area α by the sum of theforeground components will be derived from the previous frame or thesubsequent frames.

The pixel value Mc of the seventh pixel from left of the frame #n ofFIG. 41 can be represented by the following equation (19):

$\begin{matrix}{M_{c} = {{{\frac{2}{v} \cdot B}\; 06} + {\sum\limits_{i = 11}^{12}{{Fi}/v}}}} & (19)\end{matrix}$where 2/v of the first term of the right side corresponds to the mixingratio α. By exploiting the pixel value of the subsequent frame #n+1, thesecond term of the right side of the equation (19) may be represented bythe equation (20):

$\begin{matrix}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\beta \cdot {\sum\limits_{i = 7}^{110}{{Fi}/{v.}}}}} & (20)\end{matrix}$

It is here assumed, by exploiting the spatial correlation of theforeground components, the following equation (21) holds:F=F05=F06=F07=F08=F09=F10=F11=F12  (21)which may be used to rewrite the equation (20) to

$\begin{matrix}\begin{matrix}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\frac{2}{v} \cdot F}} \\{= {\beta \cdot \frac{4}{v} \cdot {F.}}}\end{matrix} & (22)\end{matrix}$

As a result, β can be represented by the following equation (23)β=22/4  (23).

In general, if it is assumed that the foreground components relevant tothe mixed area are equal, as shown by the equation (21), the followingequation (24):β=1·α  (24)holds, by the ratio of the internal division, for the totality of pixelsof the mixed area.

If the equation (24) holds, the equation (17) can be expanded as in theequation (25):

$\begin{matrix}\begin{matrix}{C = {{\alpha \cdot P} + f}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {N.}}}}\end{matrix} & (25)\end{matrix}$

Similarly, if the equation (24) holds, the equation (18) can be expandedas in the equation (26):

$\begin{matrix}\begin{matrix}{C = {{\alpha \cdot N} + f}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {P.}}}}\end{matrix} & (26)\end{matrix}$

In the equations (25) and (26), since C, N and P are known pixel values,the there is only one variable contained in the equations (25) and (26),that is the mixing ratio α. The relation among C, N and P in theequations (25) and (26) is shown in FIG. 42. It is noted that C, N and Pare a pixel value of a pixel of the frame #n under consideration, apixel value of a pixel of the frame #n+1, the position of which in thespatial direction is in register with that of the considered pixel, anda pixel value of the pixel of the frame #n+1, the position of which inthe spatial direction is in register with that of the considered pixel,respectively.

Thus, each one variable is contained in each of the equations (25) and(26), so the mixing ratio α can be calculated by exploiting the pixelvalues of the pixels of the three frames. The condition for the correctmixing ratio α to be calculated by solving the equations (25) and (26)is that the foreground components relevant to the mixed area are equal,that is that the pixel values of a number of the consecutive pixelstwice the movement quantity x, which pixels are in the picture object ofthe foreground imaged in a standstill state, and which are positioned ata boundary of the picture object in association with the movingdirection of the foreground are constant.

The mixing ratio α of the pixels belonging to the covered backgroundarea is calculated by the equation (27), whilst the mixing ratio α ofthe pixel belonging to the uncovered background area is calculated bythe following equations (27) and (28):α=(C·N)/(P·N)  (27)α=(C·P)/(N·P)  (28).

In FIG. 43, which is a block diagram showing the structure of theestimated mixing ratio processor 201, a frame memory 221 stores theinput pictures on the frame basis, and feeds a frame, next to the framebeing input as an input picture, to a frame memory 222 and to a mixingratio calculating unit 223.

The frame memory 222 stores the input pictures on the frame basis androutes a frame next following the frame being supplied from the framememory 221 to the mixing ratio calculating unit 223.

So, if the frame #n+1 is being input as an input picture to the mixingratio calculating unit 223, the frame memory 221 routes the frame #n tothe mixing ratio calculating unit 223, whilst the frame memory 222routes the frame #n·1 to the mixing ratio calculating unit 223.

The mixing ratio calculating unit 223 calculates an estimated mixingratio of the considered pixel, by calculations of the equation (27) onthe pixel value C of the pixel of the frame #n under consideration, thepixel value of the pixel of the frame #n+1, the spatial position ofwhich is in registration with that of the considered pixel, and thepixel value of the pixel of the frame #n·1, the spatial position ofwhich is in registration with that of the considered pixel, and outputsthe so-calculated estimated mixing ratio. For example, if the backgroundis at a standstill, the mixing ratio calculating unit 223 calculates theestimated mixing ratio of the considered pixel, from the pixel value Cof the pixel of the frame #n under consideration, the pixel value N ofthe pixel of the frame #n+1, the position of which in the frame is thesame as that of the considered pixel, and the pixel value P of the pixelof the frame #n·1, the position of which in the frame is the same asthat of the considered pixel, and outputs the so-calculated estimatedmixing ratio.

In this manner, the estimated mixing ratio processor 201 calculates theestimated mixing ratio, based on the input picture, to route theso-calculated estimated mixing ratio to the mixing ratio decision unit203.

The estimated mixing ratio processor 202 is similar to the estimatedmixing ratio processor 201 except that the estimated mixing ratioprocessor 201 calculates the estimated mixing ratio of the consideredpixel in accordance with the equation (27), whilst the estimated mixingratio processor 202 calculates the estimated mixing ratio of theconsidered pixel in accordance with the equation (28), and hence thecorresponding description is omitted for clarity.

FIG. 44 shows an example of the estimated mixing ratio calculated by theestimated mixing ratio processor 201. FIG. 44 shows the estimated mixingratio for the movement quantity v of the foreground corresponding to anobject moving at a constant speed equal to 11 for one line.

It is seen that the estimated mixing ratio is changing in the mixed areasubstantially linearly as shown in FIG. 38.

Reverting to FIG. 37, the mixing ratio decision unit 203 sets the mixingratio α based on the area information from the area specifying unit 103indicating to which of the foreground area, background area, coveredbackground area and the uncovered background area belongs the pixelsupplied from the area specifying unit 103 as basis for calculation ofthe mixing ratio α. The mixing ratio decision unit 203 sets 0 or 1 asthe mixing ratio if the pixel as a basis for calculation belongs to theforeground area or to the background area, respectively. On the otherhand, the mixing ratio decision unit 203 sets the estimated mixing ratiosupplied from the estimated mixing ratio processor 201 as the mixingratio α if the pixel as a basis for calculation belongs to the coveredbackground area, while setting the estimated mixing ratio supplied fromthe estimated mixing ratio processor 202 as the mixing ratio α if thepixel as a basis for calculation belongs to the uncovered backgroundarea. The e203 outputs the mixing ratio α as set based on the areainformation.

In FIG. 45, which is a block diagram showing an alternative structure ofthe mixing ratio calculating unit 104, a selection unit 231 routes thepixel belonging to the covered background area and pixels of theassociated previous and subsequent frames to an estimated mixing ratioprocessor 232, based on the area information supplied from the areaspecifying unit 103. The selection unit 231 routes the pixels belongingto the uncovered background area and pixels of the associated previousand subsequent frames to an estimated mixing ratio processor 233, basedon the area information supplied from the area specifying unit 103.

The estimated mixing ratio processor 232 calculates the estimated mixingratio of the considered pixel belonging to the covered background area,by calculations in accordance with the equation (27), based on the pixelvalues input from the selection unit 231, to route the so-calculatedestimated mixing ratio to a selection unit 234.

The estimated mixing ratio processor 233 calculates the estimated mixingratio of the considered pixel belonging to the uncovered backgroundarea, by calculations in accordance with the equation (28), based on thepixel values input from the selection unit 231, to route theso-calculated estimated mixing ratio to a selection unit 234.

The selection unit 234 sets the mixing ratio α based on the areainformation from the area specifying unit 103 indicating to which of theforeground area, background area, covered background area and theuncovered background area belongs the pixel supplied from the areaspecifying unit 103 as basis for calculation of the mixing ratio α. Themixing ratio decision unit 203 sets 0 or 1 as the mixing ratio if thepixel as a basis for calculation belongs to the foreground area or tothe background area, respectively. On the other hand, the selection unit234 sets the estimated mixing ratio supplied from the estimated mixingratio processor 232 as the mixing ratio α if the pixel as a basis forcalculation belongs to the covered background area, while setting theestimated mixing ratio supplied from the estimated mixing ratioprocessor 233 as the mixing ratio α if the pixel as a basis forcalculation belongs to the uncovered background area. The selection unit234 outputs the mixing ratio α selected and set based on the areainformation.

The mixing ratio calculating unit 104, having a modified structure shownin FIG. 45, calculates the mixing ratio α, from one pixel of the pictureto another, to output the calculated mixing ratio α.

Referring to the flowchart of FIG. 46, the processing for calculatingthe mixing ratio α of the mixing ratio calculating unit 104, theconfiguration of which is shown in FIG. 37, is explained. At step S151,the mixing ratio calculating unit 104 acquires the area informationsupplied from the area specifying unit 103. At step S151, the mixingratio calculating unit 104 acquires the area information supplied fromthe area specifying unit 103. At step S152, the estimated mixing ratioprocessor 201 calculates the estimated mixing ratio by a modelcorresponding to the covered background area to route the so-calculatedestimated mixing ratio to the mixing ratio decision unit 203. Theprocessing for calculating the estimated mixing ratio will be explainedsubsequently in detail by referring to flowchart of FIG. 47.

At step S153, the estimated mixing ratio processor 202 calculates theestimated mixing ratio by a model corresponding to the coveredbackground area to route the so-calculated estimated mixing ratio to themixing ratio decision unit 203.

At step S154, the mixing ratio calculating unit 104 checks whether ornot the mixing ratio α has been estimated for the entire frame. If it isfound that the mixing ratio α has not been estimated for the entireframe, the program reverts to step S152 to execute the processing ofestimating the mixing ratio α for the next pixel.

If it is decided at step S154 that the mixing ratio α has been estimatedfor the entire frame, the program reverts to step S155 where the mixingratio decision unit 203 sets the mixing ratio α based on the areainformation supplied from the area specifying unit 103 and whichindicates to which of the foreground area, background area, coveredbackground area or the uncovered background area belongs the pixel. Themixing ratio decision unit 203 sets 0 or 1 as the mixing ratio if thepixel as a basis for calculation belongs to the foreground area or tothe background area, respectively. On the other hand, the mixing ratiodecision unit 203 sets the estimated mixing ratio supplied from theestimated mixing ratio processor 201 as the mixing ratio α if the pixelas a basis for calculation belongs to the covered background area, whilesetting the estimated mixing ratio supplied from the estimated mixingratio processor 202 as the mixing ratio α if the pixel as a basis forcalculation belongs to the uncovered background area. The processingthen is finished.

In this manner, the mixing ratio calculating unit 104 is able tocalculate the mixing ratio α, as a characteristic value for each pixel,based on the area information supplied from the area specifying unit 103and on the input picture.

The processing for calculating the mixing ratio α by the mixing ratiocalculating unit 104 shown in FIG. 45 is similar to that explained byreferring to the flowchart of FIG. 46 and hence is not explainedspecifically.

Referring to the flowchart of FIG. 47, the processing similar to stepS152 of FIG. 46 for estimating the mixing ratio by a model correspondingto the covered background area is explained.

At step S171, the mixing ratio calculating unit 223 acquires the pixelvalue C of the considered pixel of the frame #n from the frame memory221.

At step S172, the mixing ratio calculating unit 223 acquires the pixelvalue C of the considered pixel of the frame #n·1 from the frame memory222.

At step S173, the mixing ratio calculating unit 223 acquires the pixelvalue N of the frame #n+1, corresponding to the considered pixelcontained in the input picture.

At step S174, the mixing ratio calculating unit 223 calculates theestimated mixing ratio based on the pixel value C of the consideredpixel of the frame #n, pixel value P of the pixel of the frame #n·1 andon the pixel value N of the pixel of the frame #n+1.

At step S175, the mixing ratio calculating unit 223 checks whether ornot the processing for calculating the estimated mixing ratio has beenfinished for the entire frame. If it is decided that the processing forcalculating the estimated mixing ratio has not been finished for theentire frame, the program reverts to step S171 to repeat the processingof calculating the estimated mixing ratio for the next pixel.

If it is verified at step S175 that the processing for calculating theestimated mixing ratio has been finished for the entire frame, theprocessing is finished.

In this manner, the estimated mixing ratio processor 201 is able tocalculate the estimated mixing ratio based on the input picture.

The processing for estimating the mixing ratio by the modelcorresponding to the uncovered background area at step S153 of FIG. 46is similar to the processing exploiting the equation corresponding tothe model of the uncovered background area, as shown in the flowchart ofFIG. 47, and hence is not explained specifically.

Meanwhile, since the estimated mixing ratio processor 232 and theestimated mixing ratio processor 233, shown in FIG. 45, calculates theestimated mixing ratio by executing the processing similar to theprocessing of the flowchart of FIG. 47, the corresponding operation isomitted for simplicity.

In the foregoing explanation, it is assumed that the objectcorresponding to the background is at a standstill. However, theprocessing for finding the mixing ratio α can also be applied to a casein which a picture corresponding to the background contains themovement. For example, if a picture corresponding to the background areais moving uniformly, the estimated mixing ratio processor 201 shifts theentire picture in keeping with the movement of the background to performthe processing as if the object corresponding to the background is at astandstill. On the other hand, if the picture corresponding to thebackground area contains movements of the background which differ fromone location to another, the estimated mixing ratio processor 201selects the pixel associated with the background movement, as the pixelcorresponding to the pixel belonging to the mixed area, to execute theabove-described processing.

The structure of the mixing ratio calculating unit 104 shown in FIG. 37or 45 is merely illustrative.

It is also possible for the mixing ratio calculating unit 104 to executeonly the processing for estimating the mixing ratio by the modelcorresponding to the covered background area to output the so-calculatedestimated mixing ratio as the mixing ratio α. In this case, the mixingratio α indicates the proportion of the foreground and the backgroundfor a pixel belonging to the covered background area and for a pixelbelonging to the uncovered background area, respectively. If theabsolute value of the difference between the so-calculated mixing ratioα and 1 is calculated to set the so-calculated absolute value as themixing ratio α, the signal processor 12 is able to find the mixing ratioα indicating the proportion of the background component for the pixelbelonging to the uncovered background area.

It is also possible to execute only the processing for mixing ratioestimation by a model corresponding to the uncovered background area forthe totality of the pixels to output the so-calculated estimated mixingratio as the mixing ratio α.

The foreground/background separating unit 105 is now explained. In FIG.48 which is a block diagram showing the illustrative structure of theforeground/background separating unit 105, an input picture, fed to theforeground/background separating unit 105, is supplied to a separatingunit 251, a switch 252 and to a switch 254. The area information fromthe area specifying unit 103, specifying the information indicating thecovered background area information and the information indicating theuncovered background area information, is fed to the separating unit251, whilst the area information indicating the foreground informationand the area information indicating the background area are routed tothe switches 252, 254, respectively.

The mixing ratio α supplied from the mixing ratio calculating unit 104is sent to the separating unit 251.

The separating unit 251 separates the foreground component from theinput picture, based on the area information indicating the coveredbackground area information, the area information indicating theuncovered background area information and the mixing ratio α, andsupplies the so-separated foreground component to a synthesis unit 253,while separating the background component from the input picture toroute the so-separated background component to the synthesis unit 255.

When a pixel corresponding to the foreground is input, the switch 252 isclosed, based on the area information indicating the foreground area, toroute only the pixels corresponding to the foreground contained in theinput picture to the synthesis unit 253.

When a pixel corresponding to the background is input, the switch 254 isclosed, based on the area information indicating the background area, toroute only the pixel corresponding to the background contained in theinput picture to the synthesis unit 255.

The synthesis unit 253 synthesizes a foreground component picture, basedon the component from the separating unit 251, corresponding to theforeground, and on the pixel from the switch 252, corresponding to theforeground, to output the synthesized foreground component picture.Since the foreground area and the mixed area are not overlapped, thesynthesis unit 253 applies the processing of logical sum to thecomponent corresponding to the foreground and to the pixel correspondingto the foreground to synthesize the foreground component picture.

In the initializing processing, executed first in the processing forsynthesizing the foreground component picture, the synthesis unit 253stores a picture with all zero pixel values in an internal frame memoryto store (overwrite) the foreground component picture in the processingfor synthesizing the foreground component picture. Thus, in the pixelscorresponding to the background area, in the foreground componentpicture output by the synthesis unit 253, 0s are stored as pixel values.

The synthesis unit 255 synthesizes the background component picture,based on the components from the separating unit 251 and on the pixelsfrom the switch 254 corresponding to the background, to output thesynthesized background component picture. Since the background area andthe mixed area are not overlapped, the synthesis unit 255 applies theprocessing of logical sum to the component corresponding to thebackground and to the pixel corresponding to the background tosynthesize the background component picture.

In the initializing processing, executed first in the processing forsynthesizing the background component picture, the synthesis unit 255stores a picture with all zero pixel values in an internal frame memoryto store (overwrite) the background component picture in the processingfor synthesizing the background component picture. Thus, in the pixelscorresponding to the foreground area, in the background componentpicture output by the synthesis unit 255, 0s are stored as pixel values.

FIG. 49 shows an input picture, fed to the foreground/backgroundseparating unit 105, and the foreground component picture and thebackground component picture, output from the foreground/backgroundseparating unit 105.

FIG. 49A schematically shows the displayed picture, while FIG. 49Bdiagrammatically shows a model obtained on developing one-line pixelscomprised of pixels belonging to the foreground area, pixels belongingto the background area and the pixels in the mixed area, along the timeaxis.

Referring to FIGS. 49A and 49B, the background component picture, outputfrom the foreground/background separating unit 105, is comprised of abackground component contained in the pixels belonging to the backgroundarea and pixels belonging to the mixed area.

Referring to FIGS. 49A and 49B, the foreground component picture, outputfrom the foreground/background separating unit 105, is comprised of aforeground component contained in the pixels belonging to the foregroundarea and pixels belonging to the mixed area.

The pixel values of the pixels of the mixed area are separated by theforeground/background separating unit 105 into the background componentand the foreground’ component. The background component, thus separated,makes up the background component picture along with the pixelsbelonging to the background area. The foreground component separatedmakes up the foreground component picture along with the pixelsbelonging to the foreground area.

In this manner, the pixel values of the pixels of the foregroundcomponent picture, associated with the background area, are set to 0,while meaningful pixel values are set in the pixels corresponding to theforeground area and to the pixels corresponding to the mixed area.Similarly, the pixel values of the pixels of the background componentpicture, associated with the foreground area, are set to 0, whilemeaningful pixel values are set in the pixels corresponding to thebackground area and to the pixels corresponding to the mixed area.

The processing executed by the separating unit 251 in separating theforeground and background components from the pixels belonging to themixed area is explained.

FIG. 50 diagrammatically shows a model of a picture showing foregroundand background components of two frames including the foregroundcorresponding to an object moving from left to right in the drawing. Inthe picture model of FIG. 50, the movement quantity v of the foregroundis 4, with the number of times of the virtual splitting being 4.

In the frame #n, the leftmost pixel and fourteenth to eighteenth pixelsfrom left are composed only of the background components and belong tothe background area. In the frame #n, the second to fourth pixels fromleft include background and foreground components and belong to theuncovered background area. In the frame #n, the eleventh to thirteenthpixels from left include background and foreground components and belongto the covered background area. In the frame #n, the fifth to tenthpixels from left include only foreground components and belong to thecovered foreground area.

In the frame #n+1, the first to fifth pixels and the eighteenth pixelfrom left are composed only of the background component and belong tothe background area. In the frame #n+1, the sixth to eighth pixels fromleft include background and foreground components and belong to theuncovered background area. In the frame #n+1, the fifteenth toseventeenth pixels from left include background and foregroundcomponents and belong to the covered background area. In the frame #n+1,the ninth to fourteenth pixels from left include only foregroundcomponents and belong to the covered foreground area.

FIG. 51 illustrates the processing for separating the foregroundcomponent from the pixels belonging to the covered background area. InFIG. 51, α1 to α18 represent the mixing ratio values associated withrespective pixels of the frame #n. In FIG. 51, the fifteenth toseventeenth pixels from left belong to the covered background area.

The pixel value C15 which is the fifteenth pixel from left of the frame#n is represented by the following equation (29):

$\begin{matrix}\begin{matrix}{{C\; 15} = {{B\;{15/v}} + {F\;{09/v}} + {F\;{08/v}} + {F\;{07/v}}}} \\{= {{{{\alpha 15} \cdot B}\; 15} + {F\;{09/v}} + {F\;{08/v}} + {F\;{07/v}}}} \\{= {{{{\alpha 15} \cdot P}\; 15} + {F\;{09/v}} + {F\;{08/v}} + {F\;{07/v}}}}\end{matrix} & (29)\end{matrix}$where α15 is the mixing ratio of the fifteenth pixel from left of theframe #n and P 15 is the pixel value of the fifteenth pixel from left ofthe frame #n·1.

Based on the equation (29), the sum f15 of the foreground components ofthe fifteenth pixel of the frame #n is represented by the equation (30):

$\begin{matrix}\begin{matrix}{{f\; 15} = {{F\;{09/v}} + {F\;{08/v}} + {F\;{07/v}}}} \\{= {C\; 1\;{5 \cdot {\alpha 15} \cdot P}\; 15.}}\end{matrix} & (30)\end{matrix}$

Similarly, the sum f16 of the foreground components of the sixteenthpixel of the frame #n and the sum first wiring pattern 17 of theforeground components of the seventeenth pixel of the frame #n arerepresented by the equations (31) and (32):f16=C16·α16·P16  (31)andf17=C17·α17·P17  (32)respectively.

In this manner, the foreground components fc, contained in the pixelvalue C of the pixel belonging to the covered background area, may becalculated by the equation (33):fc=C·α·P  (33).

FIG. 52 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the uncovered background area.In FIG. 52, α1 to α18 represent the values of the mixing ratio forrespective pixels of the frame #n. In FIG. 52, the second to fourthpixels from left belong to the uncovered background area.

The pixel values C02 of the second pixel from left of the frame #n arerepresented by the equation (34):

$\begin{matrix}\begin{matrix}{{C\; 02} = {{B\;{02/v}} + {B\;{02/v}} + {B\;{02/v}} + {F\;{01/v}}}} \\{= {{{{\alpha 2} \cdot B}\; 02} + {F\;{01/v}}}} \\{= {{{{\alpha 2} \cdot N}\; 02} + {F\;{01/v}}}}\end{matrix} & (34)\end{matrix}$where α2 is a mixing ratio of the second pixel from left of the frame #nand N02 is a pixel value of the second pixel from left of the frame#n+1.

Based on the equation (34), the sum f02 of the foreground component ofthe second pixel from left of the frame #n is represented by theequation (35):f02=F01/v=C02·α2·NO2  (35)

Similarly, the sum f03 of the foreground components of the third pixelfrom left of the frame #n and the sum f04 of the foreground componentsof the fourth pixel from left of the frame #n are represented by thefollowing equations (36) and (37):f03=C03·α3·NO3  (36)andf04=C04·α4·NO4  (37)respectively.

The foreground component fu included in the pixel value C of the pixelbelonging to the uncovered background area is calculated by thefollowing equation (38):fu=C·α·N  (38)where N is a pixel value of a corresponding pixel of the next frame.

In this manner, the separating unit 251 is able to separate theforeground components and the background components from the pixelsbelonging to the mixed area, based on the information indicating thecovered background area and on the information indicating the coveredbackground area, contained in the area information, and on thepixel-based mixing ratio α.

In FIG. 53, which is a block diagram showing an illustrative structureof the separating unit 251 adapted for executing the above-describedprocessing, a picture input to the separating unit 251 is input to theframe memory 301, whilst the area information indicating the coveredbackground area and the uncovered background area, and the mixing ratioα, are input to a separating processing block 302.

The frame memory 301 stores the input pictures on the frame basis. Ifthe object of processing is the frame #n, the frame memory 301 storesthe frame #n·1, directly previous to the frame #n, frame #n and theframe #n+1 next to the frame #n.

The frame memory 301 routes the corresponding pixels of the frame #n·1,frame #n and the frame #n+1, to a separating processing block 302.

Based on the area information indicating the covered background areainformation and the uncovered background area information, and on themixing ratio α, the separating processing block 302 performscalculations, explained with reference to FIGS. 51 and 52, on the pixelvalues of the corresponding pixels of the frame #n·1, frame #n and theframe #n+1, supplied from the frame memory 301, to separate theforeground and background components from the pixels belonging to themixed area of the frame #n to route the separated components to a framememory 303.

The separating processing block 302 is made up of an uncovered areaprocessor 311, a covered area processor 312, a synthesis unit 313 and asynthesis unit 314.

The uncovered area processor 311 includes a multiplier 321 whichmultiplies the pixel value of the pixel of the frame #n+1 supplied fromthe frame memory 301 with the mixing ratio α to route the resultingproduct to the switch 322, which is closed when the pixel of the frame#n (corresponding to the pixel of the frame #n+1) supplied from theframe memory 301 is in the uncovered background area, to route the pixelvalue multiplied with the mixing ratio α sent from the multiplier 321 toan operating unit 322 and to the synthesis unit 314. The pixel value ofthe pixel of the frame #n+1 from the switch 322, multiplied by themixing ratio α, is equal to the background component of the pixel valueof the corresponding pixel of the frame #n.

An operating unit 323 subtracts the background component supplied fromthe switch 322 from the pixel value of the pixel of the frame #nsupplied from the frame memory 301 to find the foreground component. Theoperating unit 323 routes the foreground component of the pixel of theframe #n, belonging to the uncovered background area, to the synthesisunit 313.

The covered area processor 312 includes a multiplier 331 whichmultiplies the pixel value of the pixel of the frame #n·1 supplied fromthe frame memory 301 with the mixing ratio α to route the resultingproduct to the switch 322, which is closed when the pixel of the frame#n (corresponding to the pixel of the frame #n·1) supplied from theframe memory 301 is in the covered background area, to route the pixelvalue multiplied with the mixing ratio α sent from the multiplier 331 toan operating unit 333 and to the synthesis unit 314. The pixel value ofthe pixel of the frame #n·1 from the switch 332, multiplied by themixing ratio α, is equal to the background component of the pixel valueof the corresponding pixel of the frame #n.

An operating unit 333 subtracts the background component supplied fromthe switch 332 from the pixel value of the pixel of the frame #nsupplied from the frame memory 301 to find the foreground component. Theoperating unit 333 routes the foreground component of the pixel of theframe #n, belonging to the covered background area, to the synthesisunit 313.

The synthesis unit 313 synthesizes the foreground component of the pixelfrom the operating unit 323, belonging to the uncovered background area,to the foreground component of the pixel belonging to the coveredbackground area, to route the resulting sum to the frame memory 303.

The synthesis unit 314 synthesizes the background component of the pixelfrom the operating unit 323, belonging to the uncovered background area,to the background component of the pixel from the switch 332 belongingto the covered background area, to route the resulting sum to the framememory 303.

The frame memory 303 stores the foreground and background components ofthe pixels of the mixed area of the frame #n, supplied from theseparating processing block 302.

The frame memory 303 outputs the stored foreground component of thepixels of the mixed area of the frame #n and the stored backgroundcomponent of the pixels of the mixed area of the frame #n.

By exploiting the mixing ratio α as a characteristic value, theforeground and background components contained in the pixel value can beseparated completely from each other.

The synthesis unit 253 synthesizes the foreground component of thepixels of the mixed area of the frame #n, output by the separating unit251, and the pixels belonging to the foreground area, to each other, togenerate a foreground component picture. The synthesis unit 255synthesizes the background component of the pixels of the mixed area ofthe frame #n, output by the separating unit 251, and the pixelsbelonging to the background area, to each other, to generate abackground component picture.

FIG. 54 shows an example of a foreground component picture and anexample of the background component picture corresponding to the frame#n of FIG. 50.

FIG. 54A shows an example of the foreground component picturecorresponding to the frame #n of FIG. 50. The pixel values of theleftmost pixel and the fourteenth pixel from left were composed only ofthe background component before foreground/background separation, andhence are equal to 0.

The second to fourth pixels from left belonged to the uncoveredbackground area before foreground/background separation, with thebackground component being 0 and with the foreground component beingleft intact. The eleventh to thirteenth pixels from left belonged to thecovered background area before foreground/background separation, withthe background component being 0 and with the foreground component beingleft intact. The fifth to tenth pixels from left are left intact becausethese pixels are composed only of the background components.

FIG. 54B shows an example of a background component picturecorresponding to the frame #n of FIG. 50. The leftmost pixel and thefourteenth pixel from left are composed only of the background componentbefore foreground/background separation, and hence are left intact.

The second to fourth pixels from left belonged to the uncoveredbackground area before foreground/background separation, with theforeground component being 0 and with the background component beingleft intact. The eleventh to thirteenth pixels from left belonged to thecovered background area before foreground/background separation, withthe background component being 0 and with the foreground component beingleft intact. The pixel values of the fifth to tenth pixels from left areset to zero because these pixels are composed only of the foregroundcomponents.

Referring to the flowchart of FIG. 55, the processing forforeground/background separation by the foreground/background separatingunit 105 is explained. At step S201, the frame memory 301 of theseparating unit 251 acquires an input picture and stores the frame #n,to be processed for foreground/background separation, along with theprevious frame #n·1 and the subsequent frame #n+1.

At step S202, the separating processing block 302 of the separating unit251 acquires the area information supplied from the mixing ratiocalculating unit 104. At step S203, the separating processing block 302of the separating unit 251 acquires the mixing ratio α routed from themixing ratio calculating unit 104.

At step S204, the uncovered area processor 311 extracts the backgroundcomponent, based on the area information and the mixing ratio α, thepixel values of pixels belonging to the uncovered background area,supplied from the frame memory 301.

At step S205, the uncovered area processor 311 extracts the foregroundcomponent, based on the area information and the mixing ratio α, thepixel values of pixels belonging to the uncovered background area,supplied from the frame memory 301.

At step S206, the covered area processor 312 extracts the backgroundcomponent, based on the area information and the mixing ratio α, thepixel values of pixels belonging to the covered background area,supplied from the frame memory 301.

At step S207, the covered area processor 312 extracts the foregroundcomponent, based on the area information and the mixing ratio α, thepixel values of pixels belonging to the covered background area,supplied from the frame memory 301.

At step S208, the synthesis unit 313 synthesizes the foregroundcomponent, belonging to the uncovered background area, extracted by theprocessing of step S205, and the foreground component, belonging to thecovered background area, extracted by the processing of step S207. Thesynthesized foreground component is routed to the synthesis unit 253,which then synthesizes the pixels belonging to the foreground areasupplied via switch 252 and the foreground component supplied form theseparating unit 251 to generate a foreground component picture.

At step S209, the synthesis unit 314 synthesizes the backgroundcomponent, belonging to the uncovered background area, extracted by theprocessing of step S204, and the background component, belonging to thecovered background area, extracted by the processing of step S206. Thesynthesized foreground component is routed to the synthesis unit 255,which then synthesizes the pixels belonging to the foreground areasupplied via switch 254 and the background component supplied form theseparating unit 251 to generate a background component picture.

At step S210, the synthesis unit 253 outputs the foreground componentpicture. At step S211, the synthesis unit 255 outputs the backgroundcomponent picture to terminate the processing.

In this manner, the foreground/background separating unit 105 is able toseparate the foreground component and the background component from theinput picture, based on the area information and the mixing ratio α, tooutput a foreground component picture, made up only of the foregroundcomponents, and the background component picture, made up only of thebackground components.

The adjustment of the quantity of the motion blurring from theforeground component picture is explained.

In FIG. 56, which is a block diagram showing an illustrative structureof the motion blurring adjustment unit 106, the motion vector suppliedfrom the motion detection unit 102, the corresponding positioninformation and the area information supplied from the area specifyingunit 103 are routed to a processing unit decision unit 351 and to amodelling unit 352. The foreground component picture supplied from theforeground/background separating unit 105 is sent to an addition unit354.

The processing unit decision unit 351 routes the generated processingunit to the modelling unit 352, along with the motion vector, based onthe motion vector, the corresponding position information and the areainformation.

The processing unit, generated by the processing unit decision unit 351,represents consecutive pixels, beginning from a pixel corresponding tothe covered background area of the foreground component picture andextending up to a pixel corresponding to the uncovered background area,along the movement direction, or consecutive pixels, beginning from apixel corresponding to the uncovered background area and extending up toa pixel corresponding to the covered background area, along the movementdirection, as shown for example in FIG. 57. The processing unit is madeup e.g., of an upper left point and a lower right point. The upper leftpoint is the position of a pixel specified by the processing unit andlying at the leftmost or uppermost point on a picture.

The modelling unit 352 executes the modelling based on the motion vectorand on the input processing unit. More specifically, the modelling unit352 may hold at the outset plural models corresponding to the number ofpixels contained in the processing unit, the number of times of thevirtual splitting of the pixel values in the time axis direction and thenumber of the pixel-based foreground components to select a modelspecifying the correspondence between the pixel values and theforeground components, based on the processing unit and on the number oftimes of the virtual splitting of pixel values in the time axisdirection, as shown in FIG. 58.

For example, with the number of pixels corresponding to a processingunit being 12 and with the movement quantity v in the shutter time being5, the modelling unit 352 selects a sum total of eight foregroundcomponents, with the leftmost pixel including one foreground component,the second left pixel including two foreground components, the thirdleft pixel including three foreground components, the fourth left pixelincluding four foreground components, the second left pixel includingtwo foreground components, the third left pixel including threeforeground components, the fourth left pixel including four foregroundcomponents, the fifth left pixel including five foreground components,the sixth left pixel including five foreground components, the seventhleft pixel including five foreground components, the eighth left pixelincluding five foreground components, the ninth left pixel includingfour foreground components, the tenth left pixel including threeforeground components, the eleventh left pixel including two foregroundcomponents, and the twelfth left pixel including one foregroundcomponent.

Instead of selecting from the pre-stored model, the modelling unit 352may generate the model based on the motion vector and on the processingunit, when the motion vector and the processing unit are suppliedthereto.

The modelling unit 352 sends a selected model to an equation generatingunit 353.

The equation generating unit 353 generates an equation based on themodel supplied from the modelling unit 352. Referring to the model ofthe foreground component picture, shown in FIG. 58, an equationgenerated by the equation generating unit 353 when the number of theforeground components is 8, the number of pixels corresponding to theprocessing unit is 12, the movement quantity v is 5, and the number oftimes of the virtual splitting is 5 is explained.

When the foreground components corresponding to the shutter time/vcontained in the foreground component picture are F01/v to F08/v, therelation between the foreground components F01/v to F08/v and the pixelvalues C01 to C12 is represented by the equations (39) to (50):C01=F01/v  (39)C02=F02/v+F01/v  (40)C03=F03/v+F02/v+F01/v  (41)C04=F04/v+F03/v+F02/v+F01/v  (42)C05=F05/v+F04/v+F03/v+F02/v+F01/v  (43)C06=F06/v+F05/v+F04/v+F03/v+F02/v  (44)C07=F07/v+F06/v+F05/v+F04/v+F03/v  (45)C08=F08/v+F07/v+F06/v+F05/v+F04/v  (46)C09=F08/v+F07/v+F06/v+F05/v  (47)C10=F08/v+F07/v+F06/v  (48)C11=F08/v+F07/v  (49)C12=F08/v  (50)

The equation generating unit 353 modifies the generated equations togenerate equations. The equations generated by the equation generatingunit 353 are indicated by the equations (51) to (62):C01=1·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (51)C02=1·F01/v+1·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (52)C03=1·F01/v+1·F02/v+1·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (53)C04=1·F01/v+1·F02/v+1·F03/v+1·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (54)C05=1·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+0·F06/v+0·F07/v+0·F08/v  (55)C06=0·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+0·F07/v+0·F08/v  (56)C07=0·F01/v+0·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+0·F08/v  (57)C08=0·F01/v+0·F02/v+0·F03/v+1·F04/v+1·F05/v+0·F06/v+1·F07/v+1·F08/v  (58)C09=0·F01/v+0·F02/v+0·F03/v+0·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (59)C10=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+1·F06/v+1·F07/v+1·F08/v  (60)C11=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+1·F08/v  (61)C12=0·F01/v+0·F02/v+0·F03/v+0·F04/v+1·F05/v+0·F06/v+1·F07/v+1·F08/v  (62)

The equations (51) to (62) may also be represented by the equation (63):

$\begin{matrix}{{Cj} = {\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {{Fi}/v}}}}} & (63)\end{matrix}$where j indicates the pixel position. In this case, j assumes one ofvalues of 1 to 12. On the other hand, i denotes the position of theforeground value, and assumes one of values of 1 to 8. Aij has values of0 or 1 in association with the values of i and j.

If an error is taken into account, the equation (63) may be representedby the equation (64):

$\begin{matrix}{{Cj} = {{\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {{Fi}/v}}}} + {ej}}} & (64)\end{matrix}$where ej is an error contained in a considered pixel C_(j).

The equation (64) may be rewritten to the equation (65):

$\begin{matrix}{{ej} = {{Cj} - {\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {{Fi}/{v.}}}}}}} & (65)\end{matrix}$

In order to apply the least square sum, a square sum of errors E isdefined as indicated by the equation (66):

$\begin{matrix}{E = {\sum\limits_{j = 01}^{12}{{ej}^{2}.}}} & (66)\end{matrix}$

In order to minimize the error, it suffices if the value of the partialdifferentiation by a variable FK with respect to the error square sum Eis 0. Fk is found to satisfy the equation (67):

$\begin{matrix}\begin{matrix}{\frac{\delta\; E}{\delta\;{Fk}} = {2 \cdot {\sum\limits_{i = 01}^{12}{{ej} \cdot \frac{\delta\;{ej}}{\delta\;{Fk}}}}}} \\{= {2 \cdot {\sum\limits_{j = 01}^{12}\left\{ {{\left( {{Cj} - {\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {{Fi}/v}}}}} \right) \cdot \left( {{- \alpha}\;{{kj}/v}} \right)} = 0.} \right.}}}\end{matrix} & (67)\end{matrix}$

Since the movement quantity v in the equation (67) is constant, theequation (68):

$\begin{matrix}{{\sum\limits_{j = 01}^{12}{\alpha\;{{kj} \cdot \left( {{Cj} - {\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {{Fi}/v}}}}} \right)}}} = 0} & (68)\end{matrix}$can be derived.

Developing the equation (68) and shifting the term, we obtain theequation (69):

$\begin{matrix}{{\sum\limits_{j = 01}^{12}\left( {\alpha\;{{kj} \cdot {\sum\limits_{i = 01}^{08}{\alpha\;{{ij} \cdot {Fi}}}}}} \right)} = {v \cdot {\sum\limits_{j = 01}^{12}{{akj} \cdot {{Cj}.}}}}} & (69)\end{matrix}$

The equation (69) is expanded into eight equations by substituting oneof integers of 1 to 8. The resulting eight equations can be representedby matrix by a sole equation termed a normal equation.

An example of the normal equation, generated by the equation generatingunit 353 based on the minimum square method, is the following equation(70):

$\begin{matrix}{{\begin{bmatrix}54321000 \\45432100 \\34543210 \\23454321 \\12345432 \\01234543 \\00123454 \\00012345\end{bmatrix}\begin{bmatrix}{F\; 01} \\{F\; 02} \\{F\; 03} \\{F\; 04} \\{F\; 05} \\{F\; 06} \\{F\; 07} \\{F\; 08}\end{bmatrix}} = {v \cdot {\begin{bmatrix}{\sum\limits_{i = 08}^{12}{Ci}} \\{\sum\limits_{i = 07}^{11}{Ci}} \\{\sum\limits_{i = 06}^{10}{Ci}} \\{\sum\limits_{i = 05}^{09}{Ci}} \\{\sum\limits_{i = 04}^{08}{Ci}} \\{\sum\limits_{i = 03}^{07}{Ci}} \\{\sum\limits_{i = 02}^{06}{Ci}} \\{\sum\limits_{i = 01}^{05}{Ci}}\end{bmatrix}.}}} & (70)\end{matrix}$

If the equation (70) is expressed as A·F=v·C, C, A and v are known,while F is unknown. On the other hand, A and v are known at themodelling stage, C becomes known on inputting a pixel value in theadding operation.

By calculating the foreground component by the normal equation which isbased on the least square method, it is possible to effect scattering oferrors contained in the pixel C.

The equation generating unit 353 sends the so-generated normal equationto an addition unit 354.

Based on the processing unit, supplied from the processing unit decisionunit 351, the addition unit 354 sets the pixel value C, contained in theforeground component picture, in the matrix equation supplied from theequation generating unit 353. The addition unit 354 sends the matrix,having set the pixel value C set therein, to an operating unit 355.

The operating unit 355 calculates the foreground component freed ofmotion blurring Fi/v, by the processing which is based on the solutionmethod, such as Gauss•JordAn erasure method, to find Fi corresponding toone of integers 0 to 8 of i as pixel values of the foreground freed ofthe motion blurring to output the foreground component picture composedof pixel values freed of motion blurring Fi to a motion blurring addingunit 356 and to a selection unit 357.

Meanwhile, F01 to F08 are set to C03 to C10, in the foreground componentpicture freed of the motion blurring, shown in FIG. 59, in order toproduce no changes in the position of the foreground component picturerelative to the picture screen. An arbitrary position can be set.

The motion blurring adding unit 356 is able to adjust the quantity ofthe motion blurring by imparting the motion blurring adjustment quantityv′ different from the movement quantity v, for example, the motionblurring adjustment quantity v′ equal to one-half the movement quantityv, or the motion blurring adjustment quantity v′ irrelevant to themovement quantity v, to adjust the value of the motion blurringquantity. For example, the motion blurring adding unit 356 divides thepixel value Fi of the foreground freed of the motion blurring by themotion blurring adjustment value v′ to calculate the foregroundcomponent Fi/v′ and sums the foreground components Fi/v′ to generate apixel value adjusted for the motion blurring quantity, as shown in FIG.60. For example, if the motion blurring adjustment quantity v′ is 3, thepixel value C02 is (F01/v′), the pixel value C03 is (F01+F02)/v′, thepixel value C04 is (F01+F02+F03)/v′ and the pixel value C05 is(F02+F03+F04)/v′.

The motion blurring adding unit 356 sends the foreground componentpicture, adjusted for the motion blurring quantity, to the selectionunit 357.

Based on the selection signal corresponding to the user's selection, theselection unit 357 selects one of the foreground component picture freedof the motion blurring, sent from the operating unit 355 and theforeground component picture from the motion blurring adding unit 356adjusted for the motion blurring quantity to output the selectedforeground component picture.

The motion blurring adjustment unit 106 thus is able to adjust themotion blurring quantity based on the selection signals and the motionblurring adjustment quantity v′.

For example, if the number of pixels associated with the selectionsignal is eight and the movement quantity v is four, as shown in FIG.61, the motion blurring adjustment unit 106 is able to generate thematrix equation (71):

$\begin{matrix}{{\begin{bmatrix}43210 \\34321 \\23432 \\12343 \\01234\end{bmatrix}\begin{bmatrix}{F\; 01} \\{F\; 02} \\{F\; 03} \\{F\; 04} \\{F\; 05}\end{bmatrix}} = {v \cdot {\begin{bmatrix}{\sum\limits_{i = 05}^{08}{Ci}} \\{\sum\limits_{i = 04}^{07}{Ci}} \\{\sum\limits_{i = 03}^{06}{Ci}} \\{\sum\limits_{i = 02}^{05}{Ci}} \\{\sum\limits_{i = 01}^{04}{Ci}}\end{bmatrix}.}}} & (71)\end{matrix}$

The motion blurring adjustment unit 106 thus establishes a number ofequations corresponding to the length of the processing unit tocalculate the pixel value Fi adjusted for the motion blurring quantity.In similar manner, if the number of pixels contained in a processingunit is 100, the motion blurring adjustment unit 106 generates 100equations in association with the 100 pixels to calculate Fi.

In FIG. 62, showing another configuration of the motion blurringadjustment unit 106, the parts or components similar to those shown inFIG. 56 are indicated by the same reference numerals and are notexplained specifically.

A selection unit 361 routes the input motion vector and thecorresponding position signal directly to the processing unit decisionunit 351 and to the modelling unit 352. Alternatively, the selectionunit 361 substitutes the motion blurring adjustment quantity v′ for themagnitude of the motion vector to route the motion vector, the magnitudeof has been replaced by the motion blurring adjustment quantity v′ andthe corresponding position signal directly to the processing unitdecision unit 351 and to the modelling unit 352.

By so doing, the processing unit decision units 351 to 355 of the motionblurring adjustment unit 106 of FIG. 62 are able to adjust the motionblurring quantity in association with the movement quantity v and withthe motion blurring adjustment quantity v′. For example, if the movementquantity v is 5 and the motion blurring adjustment quantity v′ is 3, theprocessing unit decision units 351 to 355 of the motion blurringadjustment unit 106 of FIG. 62 executes the processing on the foregroundcomponent picture, with the movement quantity v of FIG. 58 equal to 5,in accordance with the model shown in FIG. 60 corresponding to themotion blurring adjustment quantity v′ equal to 3, to calculate apicture containing the motion blurring corresponding to the movementquantity v of (movement quantity v)/(motion blurring adjustment quantityv′)=5/3, that is approximately 1.7. Since the calculated picture is freeof the motion blurring corresponding to the movement quantity v equal to3, attention is to be directed to the fact that the relation between themovement quantity v and the motion blurring adjustment quantity v′ has adifferent meaning from the results of the motion blurring adding unit356.

The motion blurring adjustment unit 106 generates an equation inassociation with the movement quantity v and the processing unit andsets the pixel values of the foreground component picture in thegenerated equation to calculate the foreground component pictureadjusted for the motion blurring quantity.

Referring to the flowchart of FIG. 63, the processing for adjusting themotion blurring quantity in the foreground component picture by themotion blurring adjustment unit 106 is explained.

At step S251, the processing unit decision unit 351 of the motionblurring adjustment unit 106 generates a processing unit, based on themotion vector and the area information, to send the generated processingunit to the modelling unit 352.

At step S252, the modelling unit 352 of the motion blurring adjustmentunit 106 selects and generates a model in association with the movementquantity v and the processing unit. At step S253, the equationgenerating unit 353 generates the normal equation, based on the selectedmodel.

At step S254, the addition unit 354 sets pixel values of the foregroundcomponent picture in the so-generated normal equation. At step S255, theaddition unit 354 verifies whether or not the pixel values of thetotality of pixels of the processing unit have been set. If it isverified that the pixel values of the totality of pixels correspondingto the processing unit have not been set, the program reverts to stepS254 to repeat the processing of setting pixel values in the normalequation.

If it is decided at step S255 that the pixel values of the totality ofpixels corresponding to the processing unit have been set, the programreverts to step S256 where the operating unit 355 calculates the pixelvalues of the foreground, adjusted for the motion blurring quantity,based on the normal equation from the addition unit 354, in which thepixel values have been set, to terminate the processing.

In this manner, the motion blurring adjustment unit 106 is able toadjust the motion blurring quantity based on the motion vector and thearea information, from the foreground component picture containing themotion blurring.

That is, the motion blurring adjustment unit 106 is able to adjust themotion blurring quantity in the pixel values as sampling data.

Meanwhile, the structure of the motion blurring adjustment unit 106,shown in FIG. 56, is merely exemplary and is not intended to limit thepresent invention.

The signal processor 12, the configuration of which is shown in FIG. 10,is able to adjust the quantity of the motion blurring contained in theinput picture. The signal processor 12, the configuration of which isshown in FIG. 10, is able to calculate the mixing ratio α as the buriedinformation to output the so-calculated mixing ratio α.

FIG. 64 is a block diagram showing a modified configuration of thefunctions of the signal processor 12.

The parts or components similar to those of FIG. 10 are indicated by thesame reference numerals and are not explained specifically.

The area specifying unit 103 sends the area information to the mixingratio calculating unit 104 and to the synthesis unit 371.

The mixing ratio calculating unit 104 sends the area information to theforeground/background separating unit 105 and to the synthesis unit 371.

The foreground/background separating unit 105 sends the foregroundcomponent picture to the synthesis unit 371.

Based on the mixing ratio α supplied from the mixing ratio calculatingunit 104 and on the area information supplied from the area specifyingunit 103, the synthesis unit 371 synthesizes an optional backgroundpicture and the foreground component picture supplied from theforeground/background separating unit 105 to output a picturesynthesized from the optional background picture and the foregroundcomponent picture.

FIG. 65 shows the configuration of the synthesis unit 371. A backgroundcomponent generating unit 381 generates a background component picture,based on the mixing ratio α and on an optional background picture, toroute the so-generated background component picture to a mixed areapicture synthesis unit 382.

The mixed area picture synthesis unit 382 synthesizes the backgroundcomponent picture supplied from the supplied from the backgroundcomponent generating unit 381 and the foreground component picture togenerate a mixed area synthesized picture which is routed to a picturesynthesis unit 383.

Based on the area information, the picture synthesis unit 383synthesizes the foreground component picture, mixed area synthesizedpicture supplied from the mixed area picture synthesis unit 382 and anoptional background picture to generate and output a synthesizedpicture.

In this manner, the synthesis unit 371 is able to synthesize theforeground component picture to an optional background picture.

The picture obtained on synthesis of a foreground component picture withan optional background picture, based on the mixing ratio α, as acharacteristic value, is more spontaneous than a picture obtained onsimply synthesizing the pixels.

FIG. 66 shows, in a block diagram, a further configuration of thefunction of the signal processor 12 adapted for adjusting the motionblurring quantity. The signal processor 12 shown in FIG. 10 calculatesthe mixing ratio α and specifies the area sequentially, whereas thesignal processor 12 shown in FIG. 66 specifies the area and calculatesthe mixing ratio α by parallel processing.

The functions similar to those shown in the block diagram of FIG. 10 aredenoted by the same reference numerals and are not explainedspecifically.

The input picture is sent to a mixing ratio calculating unit 401,foreground/background separating unit 402, an area specifying unit 103and to an object extraction unit 101.

Based on the input picture, the mixing ratio calculating unit 401calculates, for each of the pixels contained in the input picture, theestimated mixing ratio in case the pixel is assumed to belong to thecovered background area, and the estimated mixing ratio in case thepixel is assumed to belong to the uncovered background area, to supplythe so-calculated estimated mixing ratio in case the pixel is assumed tobelong to the covered background area and estimated mixing ratio in casethe pixel is assumed to belong to the uncovered background area, to theforeground/background separating unit 402.

FIG. 67 shows, in a block diagram an illustrative structure of themixing ratio calculating unit 401.

The estimated mixing ratio processor 201, shown in FIG. 67, is similarto the estimated mixing ratio processor 201 shown in FIG. 37. Theestimated mixing ratio processing unit 202, shown in FIG. 67, is thesame as the estimated mixing ratio processing unit 202 shown in FIG. 37.

The estimated mixing ratio processor 201 calculates the estimated mixingratio, from pixel to pixel, by calculations corresponding to the modelof the covered background area, based on the input picture, to outputthe so-calculated estimated mixing ratio.

The estimated mixing ratio processor 202 calculates the estimated mixingratio, from pixel to pixel, by calculations corresponding to the modelof the uncovered background area, based on the input picture, to outputthe so-calculated estimated mixing ratio.

Based on the estimated mixing ratio in case the pixel is assumed tobelong to the covered background area, and the estimated mixing ratio incase the pixel is assumed to belong to the uncovered background area,supplied from the mixing ratio calculating unit 401, and on the areainformation, supplied from the area specifying unit 103, theforeground/background separating unit 402 generates a foregroundcomponent picture from the input picture, to route the so-generatedforeground component picture to the motion blurring adjustment unit 106and to the selection unit 107.

FIG. 68 is a block diagram showing an illustrative structure of theforeground/background separating unit 402.

The parts or components similar to those of the foreground/backgroundseparating unit 105 shown in FIG. 48 are indicated by the same referencenumerals and not explained specifically.

Based on the area information supplied from the area specifying unit103, a selection unit 421 selects one of the estimated mixing ratio incase the pixel is assumed to belong to the covered background area, andthe estimated mixing ratio in case the pixel is assumed to belong to theuncovered background area, supplied from the mixing ratio calculatingunit 401, and routes the so-selected estimated mixing ratio as themixing ratio α to the separating unit 251.

Based on the mixing ratio α and the area information, supplied from theselection unit 421, the separating unit 251 separates the foregroundcomponents and the background components from the pixel values of pixelsbelonging to the mixed area, to send the foreground components extractedto the synthesis unit 253, as well as to send the background componentsto the synthesis unit 255.

The separating unit 251 may be configured similarly to the structureshown in FIG. 53.

The synthesis unit 253 synthesizes and outputs the foreground componentpicture. The synthesis unit 255 synthesizes and outputs the backgroundcomponent picture.

The motion blurring adjustment unit 106, shown in FIG. 66, may beconfigured as in FIG. 10. Based on the area information and the motionvector, the motion blurring adjustment unit 106 adjusts the quantity ofthe motion blurring supplied from the foreground/background separatingunit 402, to output the foreground component picture adjusted for themotion blurring quantity.

Based on the selection signal, corresponding to the selection by theuser, the selection unit 107 selects one of the foreground componentpicture supplied from the foreground/background separating unit 402 andthe foreground component picture from the motion blurring adjustmentunit 106, adjusted for the motion blurring quantity, to output theselected foreground component picture.

In this manner, the signal processor 12, the configuration of which isshown in FIG. 66, is able to adjust a picture, corresponding to anobject of the foreground object contained in the input picture, tooutput the resulting picture. The signal processor 12, the configurationof which is shown in FIG. 66, is able to calculate the mixing ratio α,as the buried information, as in the first embodiment, to output theso-calculated mixing ratio α.

FIG. 69 is a block diagram showing a modification of the function of thesignal processor 12 adapted for synthesizing the foreground componentpicture to an optional background picture. The signal processor 12,shown in FIG. 64, performs area identification and the calculation ofthe mixing ratio α in series, whereas the signal processor 12, shown inFIG. 69, performs area identification and the calculation of the mixingratio α in parallel.

The functions similar to those shown in the block diagram of FIG. 66 aredenoted by the same reference numerals and are not explainedspecifically.

Based on the input picture, the mixing ratio calculating unit 401, shownin FIG. 69, calculates the estimated mixing ratio for when the pixel isassumed to belong to the covered background area and the estimatedmixing ratio for when the pixel is assumed to belong to the uncoveredbackground area, for each of the pixels contained in the input picture,to route the estimated mixing ratio for when the pixel is assumed tobelong to the covered background area and the estimated mixing ratio forwhen the pixel is assumed to belong to the uncovered background area, tothe foreground/background separating unit 402 and to the synthesis unit431.

Based on the estimated mixing ratio for when the pixel is assumed tobelong to the covered background area, the estimated mixing ratio forwhen the pixel is assumed to belong to the uncovered background area,supplied from the mixing ratio calculating unit 401, and on the areainformation supplied from the area specifying unit 103, theforeground/background separating unit 402, shown in FIG. 69, generatesthe foreground component picture from the input picture to route thegenerated foreground component picture to the synthesis unit 431.

Based on the estimated mixing ratio for when the pixel is assumed tobelong to the covered background area, the estimated mixing ratio forwhen the pixel is assumed to belong to the uncovered background area,supplied from the mixing ratio calculating unit 401, and on the areainformation supplied from the area specifying unit 103, the synthesisunit 431 synthesizes an optional background area and a foregroundcomponent picture supplied from the foreground/background separatingunit 402, to output a picture synthesized from the optional backgroundarea and the foreground component picture.

FIG. 70 shows the configuration of the synthesis unit 431. The functionssimilar to those shown in the block diagram of FIG. 65 are denoted bythe same reference numerals and are not explained specifically.

Based on the area information, supplied from the area specifying unit103, a selection unit 441 selects one of the estimated mixing ratio forwhen the pixel is assumed to belong to the covered background area andthe estimated mixing ratio for when the pixel is assumed to belong tothe uncovered background area, supplied from the mixing ratiocalculating unit 401, to route the selected estimated mixing ratio asthe mixing ratio α to the background component generating unit 381.

Based on the mixing ratio α supplied from the selection unit 441 and theoptional background component picture, the background componentgenerating unit 381, shown in FIG. 70, generates a background componentpicture, to route the generated picture to the mixed area picturesynthesis unit 382.

The mixed area picture synthesis unit 382, shown in FIG. 70, synthesizesthe background component picture, supplied from the background componentgenerating unit 381, to the foreground component picture, to generate amixed area synthesized picture, which is routed to the picture synthesisunit 383.

Based on the area information, the picture synthesis unit 383synthesizes foreground component picture, the mixed area synthesizedpicture, supplied from the mixed area picture synthesis unit 382 and anoptional background picture, to generate and output a synthesizedpicture.

In this manner, the synthesis unit 431 is able to synthesize theforeground component picture to an optional background picture.

Although the mixing ratio α has been explained as a proportion of thebackground component contained in the pixel value, it may also be aproportion of the foreground component contained in the pixel value.

Although the direction of movement of the object as the foreground hasbeen explained as being from left to right, this direction is, ofcourse, not limitative.

An embodiment in which the amount of the motion blurring quantitycontained in temperature or pressure data by the similar processing asthat performed by the signal processor 12 is explained.

FIG. 71 shows an illustrative structure of a signal processing apparatusaccording to the present invention. A thermography device 451 detects IRrays, radiated from an object being measured from an enclosed IR sensor,such as an IR CCD, to generate a signal corresponding to the wavelengthor intensity of the detected IR rays. The thermography device 451analog/digital converts the generated signal to compare the convertedsignal to reference data corresponding to the reference temperature togenerate temperature data indicating the temperature of various sites ofthe object to output the generated temperature data to the signalprocessor 452.

Similarly to the sensor 11, the thermography device 451 has integratingeffects with respect to the space and time.

The temperature data the thermography device 451 routes to the signalprocessor 452 is configured similarly to the picture data of the movingpicture, and is such data in which the values indicating the temperatureof respective sites of the object being measured (corresponding to thepixel values of the picture data) are arrayed two-dimensionally alongthe spatial direction in association with the picture data frames andalso are arrayed along the temporal direction.

The signal processor 452 adjusts the distortion contained in the inputtemperature data and which has been generated as a result of movement ofthe object being measured. For example, the signal processor 452extracts a more accurate temperature of the desired site of the objectbeing measured.

FIG. 72 is a flowchart showing the processing for adjusting the motionblurring quantity by the signal processor 452. At step S301, the signalprocessor 452 acquires temperature data in which values indicating thetemperatures for respective sites of the object being measured arearrayed two-dimensionally. Based on the temperature data, the signalprocessor 452 generates data specifying the movement.

At step S302, the signal processor 452 specifies areas of temperaturedata to a foreground area comprising only the values indicating thetemperature corresponding to a desired object, a background areacomprising only the values indicating the temperature corresponding toan object other than the desired object, and a mixed area comprising thetemperature information corresponding to the desired object and thetemperature information corresponding to the object other than thedesired object.

At step S303, the signal processor 452 checks whether or not thetemperature indicating value belongs to the temperature data. If thesignal processor 452 decides that the temperature indicating valuebelongs to the mixed area, the signal processor 452 proceeds to stepS304 to calculate the mixing ratio α by the processing similar to thatof step S102 of FIG. 27.

At step S305, the signal processor 452 separates the information of thetemperature corresponding to the object desiring temperaturemeasurement, by the processing similar to the processing of step S103 ofFIG. 27, to then proceed to step S306.

For separating the temperature information at step S305, the temperatureinformation may be converted, based on the Kirchhoff's law or the lawspecifying the relation between the object temperature and the radiatedIR rays, such as Stephen-Boltzmann law, into the energy quantity of theIR rays, emitted from the object desiring temperature measurement, toseparate the energy quantity of the converted IR rays to re-convert theseparated energy quantity into temperature. By conversion into the IRray energy prior to separation, the signal processor 452 is able toseparate the temperature information more accurately than in directseparation of the temperature information.

If, at step S303, the temperature indicating value contained in thetemperature data does not belong to the mixed area, it is unnecessary toseparate the temperature information corresponding to the objectdesiring the temperature measurement. So, the processing at steps S304and S305 are skipped so that the processing proceeds to step S306.

At step S306, the signal processor 452 generates temperature data forcausing temperature measurement to correspond to the desired object,from a value indicating the temperature belonging to the foregroundtemperature and the information on the temperature which causes thetemperature measurement to correspond to the desired object.

At step S307, the signal processor 452 generates a model correspondingto the generated temperature data by the processing similar to theprocessing at step S251.

At step S308, the signal processor 452 adjusts the quantity of themotion blurring contained in the temperature data corresponding to theobject, in need of temperature measurement, by the processing similar tothat of steps S252 to S255 of FIG. 63, based on the generated model, toterminate the processing.

In this manner, the signal processor 452 adjusts the quantity of themotion blurring contained in the temperature data generated by themovement of the object being measured to calculate the more accuratetemperature difference of respective object portions.

FIG. 73 shows an illustrative structure of a signal processing apparatusaccording to the present invention for weighing measurement. Thepressure area sensor 501 is made up of plural pressure sensors tomeasure the load per unit planar area, that is the pressure. Thepressure area sensor 501 is of a structure comprised of atwo-dimensional array on a floor surface of plural pressure sensors511-1-1 to 511-M-N. When an object 512, the weight of which is beingmeasured, is moved on the pressure area sensor 501, the pressure areasensor 501 measures the pressure applied to each of the pressure sensors511-1-1 to 511-M-N to generate weight data for each of measured rangesof the pressure sensors 511-1-1 to 511-M-N to output the generatedweight data to the signal processor 502.

The pressure sensors 511-1-1 to 511-M-N are each made up of a sensorexploiting double refraction produced when an external force is appliedto a transparent elastic material, or the so-called photoelasticity.

The pressure area sensor 501 in its entirety may be constructed by asensor exploiting the photoelasticity.

FIG. 75 illustrates the load associated with the weight of each part ofthe object 512 applied to respective ones of the pressure sensors511-1-1 to 511-M-N making up the pressure area sensor 501.

The load A corresponding to the weight of the leftmost portion of theobject 512 in FIG. 75 is applied to the pressure sensor 511-m-1. Theload b corresponding to the weight of the second left portion of theobject 512 is applied to the pressure sensor 511-m-2. The load ccorresponding to the weight of the fourth left portion of the object 512is applied to the pressure sensor 511-m-3. The load d corresponding tothe weight of the fourth left portion of the object 512 is applied tothe pressure sensor 511-m-4.

The load e corresponding to the weight of the second left portion of theobject 512 is applied to the pressure sensor 511-m-5. The load fcorresponding to the weight of the fourth left portion of the object 512is applied to the pressure sensor 511-m-6. The load g corresponding tothe weight of the fourth left portion of the object 512 is applied tothe pressure sensor 511-m-7.

The weight data output by the pressure area sensor 501 corresponds tothe arrangement of the pressure sensor 511-1-1 to 511-M-N and iscomprised of weight values arrayed two-dimensionally in the spatialdirection.

FIG. 76 illustrates typical weight data output by the pressure areasensor 501 when the object 512 is moving, with the pressure area sensor501 having integrating effects.

On the pressure sensor 511-m-1 is applied a load A corresponding to theweight of the leftmost portion of the object 512 in a unit time ofmeasurement, a value A is output as a value indicating the weightincluded in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-2 a load b corresponding to the weight of the second leftportion of the object 512, and a load d corresponding to the weight ofthe leftmost portion of the object 512, so the pressure sensor 511-m-2outputs a value A+b as a value indicating the weight comprehended in theweight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-3 a load c corresponding to the weight of the third leftportion of the object 512, a load b corresponding to the weight of thesecond left portion of the object 512 and subsequently the load Acorresponding to the weight of the second left portion of the object512, so the pressure sensor 511-m-2 outputs a value A+b+c as a valueindicating the weight comprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-4 a load d corresponding to the weight of the fourth leftportion of the object 512, a load c corresponding to the weight of thethird left portion of the object 512, a load b corresponding to theweight of the second left portion of the object 512 and subsequently theload A corresponding to the weight of the leftmost portion of the object512, so the pressure sensor 511-m-2 outputs a value A+b+c+d as a valueindicating the weight comprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-5 a load e corresponding to the weight of the fifth leftportion of the object 512, a load d corresponding to the weight of thefourth left portion of the object 512, a load c corresponding to theweight of the third left portion of the object 512 and subsequently theload A corresponding to the weight of the second left portion of theobject 512, so the pressure sensor 511-m-5 outputs a value b+c+d+e as avalue indicating the weight comprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-6 a load f corresponding to the weight of the sixth leftportion of the object 512, a load e corresponding to the weight of thefifth left portion of the object 512, a load d corresponding to theweight of the fourth left portion of the object 512 and subsequently theload c corresponding to the weight of the third left portion of theobject 512, so the pressure sensor 511-m-6 outputs a value c+d+e+f as avalue indicating the weight comprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-7 a load g corresponding to the weight of the seventh leftportion of the object 512, a load f corresponding to the weight of thesixth left portion of the object 512, a load e corresponding to theweight of the fifth left portion of the object 512 and subsequently theload d corresponding to the weight of the fourth left portion of theobject 512, so the pressure sensor 511-m-7 outputs a value d+e+f+g as avalue indicating the weight comprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-8 a load g corresponding to the weight of the seventh leftportion of the object 512, a load f corresponding to the weight of thesixth left portion of the object 512, and a load e corresponding to theweight of the fifth left portion of the object 512, so the pressuresensor 511-m-8 outputs a value e+f+g as a value indicating the weightcomprehended in the weight data.

In the unit time for measurement, there are applied to the pressuresensor 511-m-9 a load g corresponding to the weight of the seventh leftportion of the object 512, and a load f corresponding to the weight ofthe sixth left portion of the object 512, so the pressure sensor 511-m-9outputs a value f+g as a value indicating the weight comprehended in theweight data.

In the unit time for measurement, there is applied to the pressuresensor 511-m-10 a load g corresponding to the weight of the seventh leftportion of the object 512, so the pressure sensor 511-m-10 outputs avalue g as a value indicating the weight comprehended in the weightdata.

The pressure area sensor 501 outputs weight data comprised of the valueA output by the pressure sensor 511-m-1, the value A+b output by thepressure sensor 511-m-2, the value A+b+c output by the pressure sensor511-m-3, the value A+b+c+d output by the pressure sensor 511-m-4, thevalue b+c+d+e output by the pressure sensor 511-m-4, the value A+b+coutput by the pressure sensor 511-m-3, the value A+b+c+d output by thepressure sensor 511-m-4, the value b+c+d+e output by the pressure sensor511-m-5, the value c+d+e+f output by the pressure sensor 511-m-6, thevalue d+e+f+g output by the pressure sensor 511-m-7, the value e+f+goutput by the pressure sensor 511-m-8, the value f+g output by thepressure sensor 511-m-9, and the value g output by the pressure sensor511-m-10.

The signal processor 502 adjusts the distortion generated by themovement of the object 512 being measured from the weight data suppliedfrom the pressure area sensor 501. For example, the signal processor 502extracts more accurate weight of the desired sites of the object 512being measured. For example, the signal processor 502 extracts the loadsA and b to g from weight data comprised of the value A, A+b, A+b+c,A+b+c+d, b+c+d+e, c+d+e+f, d+e+f+g, e+f+g, f+g and g.

Referring to the flowchart of FIG. 77, the processing for calculatingthe load executed by the signal processor 502 is explained.

At step S401, the signal processor 502 acquires weight data output bythe pressure area sensor 501. At step S402, the signal processor 502decides, based on the weight data acquired from the pressure area sensor501, whether or not the load of the object 512 is being applied to thepressure area sensor 501. If it is decided that the load of the object512 is being applied to the pressure area sensor 501, the signalprocessor 502 proceeds to step S403 to acquire the movement of theobject 512 based on changes in the weight data.

At step S404, the signal processor 502 acquires one-line data of thepressure sensor 511 contained in the weight data along the direction ofmovement acquired by the processing at step S403.

At step S405, the signal processor 502 calculates the load correspondingto the weight of the respective portions o the object 512 to terminatethe processing. The signal processor 502 calculates the loadcorresponding to the weights of respective parts of the object 512 by aprocessing similar to the processing explained with reference to theflowchart of FIG. 63.

If, at step S402, the load of the object 512 is not applied to thepressure area sensor 501, there is no weight data to be processed, sothe processing is terminated.

In this manner, the weight measurement system is able to calculate thecorrect load corresponding to the weight of each portion of the movingobject.

The signal processor 12 generating a picture of higher resolution in thespatial direction is explained.

FIG. 78 is a block diagram showing a configuration of generating highresolution picture by increasing the number of pixels per frame, asanother function of the signal processor 12.

A frame memory 701 stores an input picture on the frame basis and sendsthe stored picture to a pixel value generator 702 and to a correlationcalculating unit 703.

The correlation calculating unit 703 calculates the correlation valuesof pixel values neighboring to one another in a transverse direction,contained in a picture supplied from the frame memory 701, to send thecalculated correlation values to the pixel value generator 702. Thepixel value generator 702 calculates double density picture componentsfrom the pixel values of the center pixel, based on the correlationvalues supplied from the correlation calculating unit 703, to generate ahorizontal double-density picture, with the so-calculated picturecomponent as the pixel value. The pixel value generator 702 sends theso-generated horizontal double-density picture to the frame memory 704.

The frame memory 704 stores the horizontal double-density picture,supplied from the pixel value generator 702, to send the so-storedhorizontal double-density picture to a pixel value generating unit 705and to a correlation calculating unit 706.

The correlation calculating unit 706 calculates the correlation valuesof pixel values neighboring to one another in a vertical direction,contained in a picture supplied from the frame memory 704, to send thecalculated correlation values to the pixel value generator 705. Thepixel value generator 705 calculates double density picture componentsfrom the pixel values of the center pixel, based on the correlationvalues supplied from the correlation calculating unit 703, to generate ahorizontal double-density picture, with the so-calculated picturecomponent as the pixel value. The pixel value generator 705 outputs theso-generated horizontal double-density picture.

The processing for generating a horizontal double-density picture by thepixel value generator 702 is explained.

FIG. 79 shows an arrangement of pixels provided in a sensor 11 as a CCD,and an area for pixel data of a horizontal double-density picture. InFIG. 79, A to I indicate individual pixels. The areas A to r each denotea light reception area obtained on halving the individual pixels A to Iin the longitudinal direction. With the width 21 of the light receptionarea of each of the pixels A to I, the width of each of the areas A to ris I. The pixel value generator 702 calculates the pixel values of thepixel data associated with the areas A to r.

FIG. 80 illustrates pixel data corresponding to the light incident onthe areas A to r. In FIG. 80, f′(x) denotes an spatially ideal pixelvalue in association with the input light and to the spatially tinydomain.

If a pixel value of a pixel data is represented by the uniformintegration of the ideal pixel value f′(x), the pixel value Y1 of pixeldata associated with the area i is represented by the equation (72):

$\begin{matrix}{{Y\; 1} = {{f(x)}{{dx} \cdot \frac{1}{e}}}} & (72)\end{matrix}$whilst the pixel value Y2 of the picture data associated with the area jand the pixel value Y3 of the pixel E are represented by the followingequations (73) and (74):

$\begin{matrix}{{{Y\; 2} = {\int_{x\; 2}^{x\; 3}{{f(x)}\ {{\mathbb{d}x} \cdot \frac{1}{e}}}}}{and}} & (73) \\\begin{matrix}{{Y\; 3} = {\int_{x\; 1}^{x\; 3}{{f(x)}\ {{\mathbb{d}x} \cdot \frac{1}{2e}}}}} \\{= \frac{{Y\; 1} + {Y\; 2}}{2}}\end{matrix} & (74)\end{matrix}$respectively.

In the above equations (72) to (74), x1, x2 and x3 are spatialcoordinates of the respective boundaries of the light reception area,area i and the area j of the pixel E, respectively.

By modifying the equation (74), the following equations (75), (76) maybe derived:Y1=2·Y3·Y2  (75)andY2=2·Y3·Y1  (76).

Therefore, if the pixel value Y3 of the pixel E and the pixel value Y2of the pixel data corresponding to the area j are known, the pixel valueY1 of the pixel data corresponding to the area i may be calculated fromthe equation (75). Also, if the pixel value Y3 of the pixel E and thepixel value Y1 of the pixel data corresponding to the area i are known,the pixel value Y2 of the pixel data corresponding to the area j can becalculated from the area j.

If the pixel value corresponding to a pixel and one of pixel values ofthe pixel data corresponding to the two areas of the pixel are known,the pixel value of the other pixel data corresponding to the other areasof the pixel may be calculated.

Referring to FIG. 81, the manner of calculating the pixel values of thepixel data corresponding to the two areas of one pixel is explained.FIG. 81A shows the relation between the pixels D, E and F and thespatially ideal pixel value f′(x).

Since the pixels D to F own integrating effects, and one pixel outputs apixel value, each one pixel value is output, as shown in FIG. 81B. Thepixel value output by the pixel E corresponds to the integrated value ofthe pixel value f′(x) in the range of the light reception area.

The correlation calculating unit 703 generates the correlation valuebetween the pixel value of the pixel D and that of the pixel E, and thecorrelation value between the pixel value of the pixel E and that of thepixel F, to route the so-generated correlation value to the pixel valuegenerator 702. The correlation value calculated by the correlationcalculating unit 703 is calculated based on the difference between thepixel value of the pixel D and that of the pixel E, or on the differencebetween the pixel value of the pixel E and that of the pixel F. When thepixel values of neighboring pixels are closer to each other, thesepixels may be said to have higher correlation. That is, a smaller valueof the difference between pixel values indicate stronger correlation.

So, if the difference between the pixel value of the pixel D and that ofthe pixel E, or the difference between the pixel value of the pixel Eand that of the pixel F, is directly used as a correlation value, thecorrelation value, which is the smaller difference value, exhibitsstronger correlation.

For example, if the correlation between the pixel value of the pixel Dand that of the pixel E are stronger than that between the pixel valueof the pixel E and that of the pixel F, the pixel value generator 702divides the pixel value of the pixel D by 2 to use the resulting valueas pixel data of the area i.

Based on the pixel value of the pixel E and on the pixel value of thearea i, the pixel value generator 702 calculates the pixel values of thepixel data of the area j in accordance with the equation (75) or (76),as shown in FIG. 81D.

The pixel value generator 702 calculates the pixel values of the pixeldata of the area g and those of the pixel data of the area h, for e.g.,the pixel D, to calculate the pixel value of the pixel data of the areai and the pixel value of the pixel data of the area j, and then tocalculate the pixel value of the pixel data of the area k and the pixelvalue of the pixel data of the area l, and so on, to calculate the pixelvalues of the pixel data in the picture as described above to generate ahorizontal double-density picture comprehending the pixel values of thepixel data calculated to furnish the so-generated horizontaldouble-density picture to the frame memory 704.

Similarly to the pixel value generator 702, the pixel value generatingunit 705 calculates, from the correlation of the pixel values of threevertically arrayed pixels of the horizontal double-density picture,supplied from the correlation calculating unit 706, and from the pixelvalues of the three pixels, the pixel values of the picture datacorresponding to two areas obtained on vertically splitting the lightreception area of the pixel, to thereby generate the double densitypicture.

When fed with the picture shown as an example in FIG. 82, the pixelvalue generating unit 702 generates a double-density picture shown as anexample in FIG. 83.

When fed with a picture, shown as an example in FIG. 82, the pixel valuegenerating unit 705 generates a picture, shown as an example in FIG. 84.When fed with a horizontal double-density picture, shown as an examplein FIG. 83, the pixel value generating unit 705 generates adouble-density picture, shown as an example in FIG. 85.

FIG. 86 is a flowchart for illustrating the processing for generatingthe double-density picture by the signal processor 12, a structure ofwhich is shown in FIG. 78. At step S601, the signal processor 12acquires an input picture to store it in the frame memory 701.

At step S602, the correlation calculating unit 703 selects one of thepixels in the picture as a considered pixel, and finds a pixelhorizontally neighboring to the considered pixel, based on the pixelvalue stored in the frame memory 701. At step S603, the pixel valuegenerator 702 generates a pixel value of pixel data lying on one side ofthe horizontal double-density picture from a pixel value exhibitingstronger correlation, that is a higher correlation value.

Based on the characteristics of the CCD, the pixel value generator 702at step S604 generates pixel values of other pixel data of thehorizontal double-density picture. Specifically, the pixel valuegenerator 702 calculates pixel values of the other picture data of thehorizontal double-density picture, based on the pixel value calculatedby the processing of step S603 and on the pixel value of the picturedata of the input picture, in accordance with the equations (75) and(76) explained with reference to FIG. 80. The picture data of thehorizontal double-density picture for the considered pixel, generated bythe processing at steps S603 and S604, are stored in the frame memory704.

At step S605, the pixel value generator 702 checks whether or not theprocessing of the entire picture has come to a close. If it isdetermined that the processing of the entire picture has come to aclose, the program reverts to step S602 to select the next pixel as theconsidered pixel to repeat the processing of generating the horizontaldouble-density picture.

If it is determined at step S605 that the processing of the entirepicture has come to a close, the correlation calculating unit 706selects one of the pixels in the picture as the considered pixel to findthe correlation value of the pixel neighboring to the considered pixelin the vertical direction based on the pixel value of the horizontaldouble-density picture stored in the frame memory 704. At step S607, thepixel value generating unit 705 generates the pixel value of the oneside of the double-density picture from the pixel values of the strongercorrelation, based on the correlation value supplied from thecorrelation calculating unit 706.

At step S608, as at step S604, the pixel value generating unit 705generates the other pixel value of the double-density picture, based onthe characteristics of the CCD. Specifically, the pixel value generator702 calculates the pixel values of the other picture data of thedouble-density picture, based on the pixel values calculated by theprocessing at step S607 and on the pixel value of the pixel data of thehorizontal double-density picture, in accordance with the equations (75)and (76) explained with reference to FIG. 80.

At step S609, the pixel value generating unit 705 decides whether or notthe processing of the entire picture has been finished. If it is decidedthat the processing of the entire picture has not been finished, theprogram reverts to step S606 to select the next pixel as the consideredpixel to repeat the processing of generating the double-density picture.

If it is decided at step S609 that the processing of the entire picturehas been finished, the pixel value generating unit 705 outputs theso-generated double-density picture to complete the processing.

In this manner, a double-density picture, the number of pixels of whichin the vertical and in the horizontal directions are doubled, may beproduced from the input picture by the signal processor 12, thestructure of which is shown in FIG. 78.

The signal processor 12, the structure of which is shown in FIG. 78, isable to generate a picture of high spatial resolution by performingsignal processing taking account of the pixel correlation and theintegrating effect of the CCD with respect to the space.

In the foregoing, a picture of the real space having a three-dimensionalspace and the time axis information is mapped on a time space having thetwo-dimensional space and the time axis information using a videocamera. The present invention is, however, not limited to thisembodiment and may be applied to correction of distortion caused byprojecting the first information of a higher order first dimension tothe lower-order second dimension, extraction of the significantinformation or to synthesis of more spontaneous pictures.

The sensor 11 may also be a sensor exemplified by, for example, a BBD(bucket brigard device), a CID (charge injection device) or a CDD(charge priming device), without being limited to a CCD. The sensor 11may also be a sensor in which detection devices are arranged in a rowinstead of in a matrix.

The recording medium, having recorded thereon a program for executingthe signal processing of the present invention may not only beconstructed by a package medium, distributed to users for furnishing theprogram separately from a computer, inclusive of a magnetic disc 51,such as a floppy disc, having the program pre-recorded thereon, anoptical disc 52, such as CD-ROM, Compact Disc, read-only memory or DVD(digital versatile disc), a magneto-optical disc 53, such as MD(mini-disc) or a semiconductor memory 54. But may also be constructed bya ROM 22 furnished to the user in a pre-assembled state in a computer,and having the program recorded thereon, and a hard disc included in thememory unit 28.

It should be noted that, in the present specification, the step forstating a program recorded on a recording medium includes not only theprocessing carried out chronologically in the specified sequence butalso the processing that is not necessarily processed chronologicallybut is executed in parallel or batch-wise.

Thus, based on the area information specifying a foreground area made uponly of foreground object components making up a foreground object inthe picture data, a background area made up only of background objectcomponents making up a background object in the picture data, and on amixed area which is a mixture of the foreground object components andthe background object components in the picture data, and on the picturedata, the mixed area including a covered background area formed at aleading end in the movement direction of the foreground object, and anuncovered background area formed at a trailing end of the foregroundobject, a processing unit made up of pixel data lying on at least astraight line extending in a direction coincident with the direction ofmovement of the foreground object from an outer end of the coveredbackground area to an outer end of the uncovered background area,centered about the foreground area, are set. A normal equation is thengenerated by setting pixel values of pixels in the processing unitdecided based on the processing unit and an unknown dividing valueobtained on dividing the foreground object components in the mixed areawith a predetermined dividing number. This normal equation is solved bythe least square method to generate foreground object componentsadjusted for the quantity of movement blurring to adjust the movementblurring quantity.

Also, sample data present in detection data lying ahead and at back ofthe considered detection data where there exist considered sample datawhich is the sample data under consideration is extracted as foregroundsample data corresponding to the foreground in the real world, whilstsample data present in detection data lying ahead and at back of theconsidered detection data where there exist considered sample data whichis the sample data under consideration is extracted as background sampledata corresponding to the background in the real world, and the mixingratio of the considered sample data is detected based on the consideredsample data, foreground sample data and on the background sample data,thereby enabling the detection of the mixing ratio.

The still/movement decision is given, based on the detection data, and amixed area containing sample data comprised of a mixture of pluralobjects in the real world is detected, thereby enabling detection of themixing ratio.

The second signal of a second dimension is acquired by detecting thefirst signal of the real world having a first dimension as mapped on thesensor, with the second dimension being lower than the first dimension,and the signal processing is performed on the second signal, therebyenabling the significant information buried due to projection to beextracted from the second signal.

Since the second signal is acquired by detecting the first signal of thereal world having the first dimension by the sensor, with the secondsignal being of a second dimension lower than the first dimension andpresenting distortion relative to the first signal, and a third signal,alleviated in distortion relative to the second signal, is generated byprocessing based on the second signal, it is possible to alleviate thesignal distortion.

In the detection signal, the foreground area, composed only of theforeground object components, constituting the foreground object, thebackground area, composed only of the background object components,constituting the background object, and the mixed area composed of theforeground object components and the background object components, arespecified, the mixing ratio of the foreground object components andbackground object components at least in the mixed area-is detected, andthe foreground object components and background object components areseparated from each other based on the specified results and on themixing ratio, thus enabling utilization of the foreground and backgroundobjects as data of higher quality.

In the detection signal, the foreground area, composed only of theforeground object components, constituting the foreground object, thebackground area, composed only of the background object components,constituting the background object, and the mixed area composed of theforeground object components and the background object components, arespecified, and the mixing ratio of the foreground and background objectcomponents in at least the mixed area is determined based on thespecified results, thus enabling detection of the mixing ratio as thesignificant information.

The mixing ratio of the foreground and background object components inthe mixed area comprised of a mixture of the foreground objectcomponents, constituting the foreground object, and the backgroundobject components, constituting the background object, is detected, andthe foreground and background object components are separated from eachother based on the mixing ratio, thus enabling utilization of theforeground and background objects as data of higher quality.

1. A signal processing apparatus for processing detection data, acquiredduring a predetermined time period by a sensor made up of apredetermined number of detection elements having time-integratingeffects, every predetermined time period, said signal processingapparatus comprising: still/movement decision means for decidingstill/movement based on said detection data; and detection means fordetecting a mixed area containing sample data having plural real worldobjects mixed together based on the results of discrimination, whereinsaid still/movement decision means includes: first decision means fordiscriminating whether said sample data being discriminated has moved,before a reference time point, from a state in which the sample valuedata is substantially constant with lapse of time to a state in whichthe sample value data is changed with lapse of time, and second decisionmeans for discriminating whether said sample data being discriminatedhas moved, after said reference time point, from a state in which thesample value data is changed with lapse of time to a state in which thesample value data is substantially constant with lapse of time; saiddetection means detecting said sample data being discriminated as saidsample data belonging to said mixed area when said first decision meansdetermining that said sample data being discriminated has moved, beforea reference time point, from a state in which the sample value data issubstantially constant with lapse of time to a state in which the samplevalue data is changed with lapse of time, or when said second decisionmeans determining that said sample data being discriminated has moved,after said reference time point, from a state in which the sample valuedata is changed with lapse of time to a state in which the sample valuedata is substantially constant with lapse of time.
 2. The signalprocessing apparatus according to claim 1 wherein when said firstdecision means determining that said sample data being discriminated hasmoved, before a reference time point, from a state in which the samplevalue data is substantially constant to a state in which the samplevalue data is changed with lapse of time with lapse of time, saiddetection means detects said sample data being discriminated as saidsample data belonging to a covered background area; and wherein whensaid second decision means determining that said sample data beingdiscriminated has moved, after a reference time point, from a state inwhich the sample value data is changed with lapse of time to a state inwhich the sample value data is substantially constant with lapse oftime, said detection means detects said sample data being discriminatedas said sample data belonging to an uncovered background area.
 3. Thesignal processing apparatus according to claim 1 wherein said detectiondata is picture data.
 4. A signal processing method for processingdetection data, acquired every predetermined time period by a sensormade up of a predetermined number of detection elements havingtime-integrating effects, every predetermined time period, said signalprocessing method comprising: a still/movement decision step of decidingstill/movement by a still/moving decision circuit based on saiddetection data; and a detection step of detecting a mixed areacontaining sample data having plural real world objects mixed togetherbased on the results of discrimination, wherein said still/movementdecision step includes: a first decision step for discriminating whethersaid sample data being discriminated has moved, before a reference timepoint, from a state in which the sample value data is substantiallyconstant with lapse of time to a state in which the sample value data ischanged with lapse of time, and a second decision step fordiscriminating whether said sample data being discriminated has moved,after said reference time point, from a state in which the sample valuedata is changed with lapse of time to a state in which the sample valuedata is substantially constant with lapse of time; said detection stepdetecting said sample data being discriminated as said sample databelonging to said mixed area when said first decision step determinesthat said sample data being discriminated has moved, before a referencetime point, from a state in which the sample value data is substantiallyconstant with lapse of time to a state in which the sample value data ischanged with lapse of time, or when said second decision step determinesthat said sample data being discriminated has moved, after saidreference time point, from a state in which the sample value data ischanged with lapse of time to a state in which the sample value data issubstantially constant with lapse of time.
 5. A signal processingprogram recorded on a computer-readable medium for processing detectiondata, acquired every predetermined time period by a sensor made up of apredetermined number of detection elements having time-integratingeffects, every predetermined time period, said signal processing programcomprising: a still/movement decision step of deciding still/movementbased on said detection data; and a detection step of detecting a mixedarea containing sample data having plural real world objects mixedtogether based on the results of discrimination, wherein saidstill/movement decision step includes: first decision step fordiscriminating whether said sample data being discriminated has moved,before a reference time point, from a state in which the sample valuedata is substantially constant with lapse of time to a state in whichthe sample value data is changed with lapse of time, and second decisionstep for discriminating whether said sample data being discriminated hasmoved, after said reference time point, from a state in which the samplevalue data is changed with lapse of time to a state in which the samplevalue data is substantially constant with lapse of time; said detectionstep detecting said sample data being discriminated as said sample databelonging to said mixed area when said first decision step determinesthat said sample data being discriminated has moved, before a referencetime point, from a state in which the sample value data is substantiallyconstant with lapse of time to a state in which the sample value data ischanged with lapse of time, or when said second decision step determinesthat said sample data being discriminated has moved, after saidreference time point, from a state in which the sample value data ischanged with lapse of time to a state in which the sample value data issubstantially constant with lapse of time.